{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOKnOyKh55aGYKEoNr2xiIj",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DanielWarfield1/MLWritingAndResearch/blob/main/RouteLLM_BT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# RouteLLM BT Model Approach\n",
        "The paper \"RouteLLM\" Suggests a few approaches to LLM Routing. Tthis is the Bradley-Terry Approach"
      ],
      "metadata": {
        "id": "cDdMPaC9uXZn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Defining Main Dataset"
      ],
      "metadata": {
        "id": "vsNVdZOfUayI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "WPYCVKHIGkNs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca975864-0a47-44fb-83a0-6c3f0860ca41"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv(\"hf://datasets/lmsys/lmsys-arena-human-preference-55k/train.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 910
        },
        "id": "vXoE4OBcrUmS",
        "outputId": "e5cecbe6-8316-401c-c13c-88516104c793"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "               id             model_a              model_b  \\\n",
              "0           30192  gpt-4-1106-preview           gpt-4-0613   \n",
              "1           53567           koala-13b           gpt-4-0613   \n",
              "2           65089  gpt-3.5-turbo-0613       mistral-medium   \n",
              "3           96401    llama-2-13b-chat  mistral-7b-instruct   \n",
              "4          198779           koala-13b   gpt-3.5-turbo-0314   \n",
              "...           ...                 ...                  ...   \n",
              "57472  4294656694          gpt-4-0613             claude-1   \n",
              "57473  4294692063          claude-2.0     llama-2-13b-chat   \n",
              "57474  4294710549            claude-1           alpaca-13b   \n",
              "57475  4294899228              palm-2       tulu-2-dpo-70b   \n",
              "57476  4294947231  gemini-pro-dev-api   gpt-4-1106-preview   \n",
              "\n",
              "                                                  prompt  \\\n",
              "0      [\"Is it morally right to try to have a certain...   \n",
              "1      [\"What is the difference between marriage lice...   \n",
              "2      [\"explain function calling. how would you call...   \n",
              "3      [\"How can I create a test set for a very rare ...   \n",
              "4      [\"What is the best way to travel from Tel-Aviv...   \n",
              "...                                                  ...   \n",
              "57472  [\"A simple mnemonic for \\u03c0:\\n\\\"How I wish ...   \n",
              "57473  [\"In python, implement a naive Bayes with gaus...   \n",
              "57474  [\"is it unethical to work on building weapons?...   \n",
              "57475  [\"If a bait contains 0,0025% bromadiolon then ...   \n",
              "57476  [\"three kids eat three apples in three days, h...   \n",
              "\n",
              "                                              response_a  \\\n",
              "0      [\"The question of whether it is morally right ...   \n",
              "1      [\"A marriage license is a legal document that ...   \n",
              "2      [\"Function calling is the process of invoking ...   \n",
              "3      [\"Creating a test set for a very rare category...   \n",
              "4      [\"The best way to travel from Tel Aviv to Jeru...   \n",
              "...                                                  ...   \n",
              "57472  [\"Sure, let's break it down:\\n\\n1. \\\"How\\\" has...   \n",
              "57473  [\"Here is an implementation of a naive Bayes c...   \n",
              "57474  [\"Working on weapons technology raises some et...   \n",
              "57475  [\"Bromadiolone is a rodenticide which is most ...   \n",
              "57476                                      [\"27 apples\"]   \n",
              "\n",
              "                                              response_b  winner_model_a  \\\n",
              "0      [\"As an AI, I don't have personal beliefs or o...               1   \n",
              "1      [\"A marriage license and a marriage certificat...               0   \n",
              "2      [\"Function calling is the process of invoking ...               0   \n",
              "3      [\"When building a classifier for a very rare c...               1   \n",
              "4      [\"The best way to travel from Tel-Aviv to Jeru...               0   \n",
              "...                                                  ...             ...   \n",
              "57472  [\"Here is how that mnemonic represents the dig...               1   \n",
              "57473  [\"Sure! Here's an implementation of a naive Ba...               1   \n",
              "57474  [\"It depends on the context. Weapons can be us...               1   \n",
              "57475  [\"As an AI language model, I do not promote or...               0   \n",
              "57476  [\"If three kids eat three apples in three days...               1   \n",
              "\n",
              "       winner_model_b  winner_tie  \n",
              "0                   0           0  \n",
              "1                   1           0  \n",
              "2                   0           1  \n",
              "3                   0           0  \n",
              "4                   1           0  \n",
              "...               ...         ...  \n",
              "57472               0           0  \n",
              "57473               0           0  \n",
              "57474               0           0  \n",
              "57475               1           0  \n",
              "57476               0           0  \n",
              "\n",
              "[57477 rows x 9 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ddb419c8-5184-4f31-a736-e5e8935a748a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>model_a</th>\n",
              "      <th>model_b</th>\n",
              "      <th>prompt</th>\n",
              "      <th>response_a</th>\n",
              "      <th>response_b</th>\n",
              "      <th>winner_model_a</th>\n",
              "      <th>winner_model_b</th>\n",
              "      <th>winner_tie</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>30192</td>\n",
              "      <td>gpt-4-1106-preview</td>\n",
              "      <td>gpt-4-0613</td>\n",
              "      <td>[\"Is it morally right to try to have a certain...</td>\n",
              "      <td>[\"The question of whether it is morally right ...</td>\n",
              "      <td>[\"As an AI, I don't have personal beliefs or o...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>53567</td>\n",
              "      <td>koala-13b</td>\n",
              "      <td>gpt-4-0613</td>\n",
              "      <td>[\"What is the difference between marriage lice...</td>\n",
              "      <td>[\"A marriage license is a legal document that ...</td>\n",
              "      <td>[\"A marriage license and a marriage certificat...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>65089</td>\n",
              "      <td>gpt-3.5-turbo-0613</td>\n",
              "      <td>mistral-medium</td>\n",
              "      <td>[\"explain function calling. how would you call...</td>\n",
              "      <td>[\"Function calling is the process of invoking ...</td>\n",
              "      <td>[\"Function calling is the process of invoking ...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>96401</td>\n",
              "      <td>llama-2-13b-chat</td>\n",
              "      <td>mistral-7b-instruct</td>\n",
              "      <td>[\"How can I create a test set for a very rare ...</td>\n",
              "      <td>[\"Creating a test set for a very rare category...</td>\n",
              "      <td>[\"When building a classifier for a very rare c...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>198779</td>\n",
              "      <td>koala-13b</td>\n",
              "      <td>gpt-3.5-turbo-0314</td>\n",
              "      <td>[\"What is the best way to travel from Tel-Aviv...</td>\n",
              "      <td>[\"The best way to travel from Tel Aviv to Jeru...</td>\n",
              "      <td>[\"The best way to travel from Tel-Aviv to Jeru...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57472</th>\n",
              "      <td>4294656694</td>\n",
              "      <td>gpt-4-0613</td>\n",
              "      <td>claude-1</td>\n",
              "      <td>[\"A simple mnemonic for \\u03c0:\\n\\\"How I wish ...</td>\n",
              "      <td>[\"Sure, let's break it down:\\n\\n1. \\\"How\\\" has...</td>\n",
              "      <td>[\"Here is how that mnemonic represents the dig...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57473</th>\n",
              "      <td>4294692063</td>\n",
              "      <td>claude-2.0</td>\n",
              "      <td>llama-2-13b-chat</td>\n",
              "      <td>[\"In python, implement a naive Bayes with gaus...</td>\n",
              "      <td>[\"Here is an implementation of a naive Bayes c...</td>\n",
              "      <td>[\"Sure! Here's an implementation of a naive Ba...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57474</th>\n",
              "      <td>4294710549</td>\n",
              "      <td>claude-1</td>\n",
              "      <td>alpaca-13b</td>\n",
              "      <td>[\"is it unethical to work on building weapons?...</td>\n",
              "      <td>[\"Working on weapons technology raises some et...</td>\n",
              "      <td>[\"It depends on the context. Weapons can be us...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57475</th>\n",
              "      <td>4294899228</td>\n",
              "      <td>palm-2</td>\n",
              "      <td>tulu-2-dpo-70b</td>\n",
              "      <td>[\"If a bait contains 0,0025% bromadiolon then ...</td>\n",
              "      <td>[\"Bromadiolone is a rodenticide which is most ...</td>\n",
              "      <td>[\"As an AI language model, I do not promote or...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57476</th>\n",
              "      <td>4294947231</td>\n",
              "      <td>gemini-pro-dev-api</td>\n",
              "      <td>gpt-4-1106-preview</td>\n",
              "      <td>[\"three kids eat three apples in three days, h...</td>\n",
              "      <td>[\"27 apples\"]</td>\n",
              "      <td>[\"If three kids eat three apples in three days...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>57477 rows × 9 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ddb419c8-5184-4f31-a736-e5e8935a748a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ddb419c8-5184-4f31-a736-e5e8935a748a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ddb419c8-5184-4f31-a736-e5e8935a748a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-013c34cd-2f19-45a8-8940-c3a008ebc392\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-013c34cd-2f19-45a8-8940-c3a008ebc392')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-013c34cd-2f19-45a8-8940-c3a008ebc392 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_5d2ff484-6d67-4672-b330-e7ae8a66975a\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_5d2ff484-6d67-4672-b330-e7ae8a66975a button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 57477,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1238326827,\n        \"min\": 30192,\n        \"max\": 4294947231,\n        \"num_unique_values\": 57477,\n        \"samples\": [\n          2785062085,\n          48259531,\n          3622731894\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"model_a\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 64,\n        \"samples\": [\n          \"mistral-7b-instruct\",\n          \"guanaco-33b\",\n          \"gpt-4-1106-preview\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"model_b\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 64,\n        \"samples\": [\n          \"llama2-70b-steerlm-chat\",\n          \"llama-13b\",\n          \"gpt-4-0613\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"prompt\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 51734,\n        \"samples\": [\n          \"[\\\"Are autoregressive language models (e.g. GPT-3) currently more popular than auto-encoding language models (e.g. BERT)? If so, what are the explanations?\\\"]\",\n          \"[\\\"Write a c++ algorithm that aims to find the rightmost zero in a binary representation, flip it to a one, and flip any 1s to its right\\\"]\",\n          \"[\\\"What is J\\\\u00e9r\\\\u00f4me WAX ?\\\"]\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response_a\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 56566,\n        \"samples\": [\n          \"[\\\"Tea is grown commercially in the Azores, an autonomous region of Portugal. The main tea-producing estates are located on the island of S\\\\u00e3o Miguel, where the only commercial tea plantations in Europe can be found. The most well-known tea estates on S\\\\u00e3o Miguel are Gorreana Tea Estate and Porto Formoso Tea Factory.\\\"]\",\n          \"[\\\"Sure! Here's an example to illustrate how function pointers work in C:\\\\n\\\\n```c\\\\n#include <stdio.h>\\\\n\\\\n\\\\/\\\\/ Function to be pointed to\\\\nint add(int a, int b) {\\\\n    return a + b;\\\\n}\\\\n\\\\n\\\\/\\\\/ Function to demonstrate function pointers\\\\nvoid functionPointerExample(int (*funcPtr)(int, int)) {\\\\n    int result = funcPtr(5, 10);\\\\n    printf(\\\\\\\"Result: %d\\\\\\\\n\\\\\\\", result);\\\\n}\\\\n\\\\nint main() {\\\\n    \\\\/\\\\/ Use the address-of operator (&) to get the address of the add function\\\\n    int (*addPtr)(int, int) = &add;\\\\n\\\\n    \\\\/\\\\/ Call the functionPointerExample function and pass the address of the add function\\\\n    functionPointerExample(addPtr);\\\\n\\\\n    \\\\/\\\\/ Call the add function directly\\\\n    int result = add(15, 20);\\\\n    printf(\\\\\\\"Result: %d\\\\\\\\n\\\\\\\", result);\\\\n\\\\n    return 0;\\\\n}\\\\n```\\\\n\\\\nIn this example, we define a function `add` that takes two integers as arguments and returns their sum.\\\\n\\\\nWe then define a function `functionPointerExample` that takes a function pointer as an argument. This function calls the pointed-to function passing it two integers, and then prints the result.\\\\n\\\\nIn the main function, we create a function pointer `addPtr` which points to the `add` function. We then call the `functionPointerExample` function passing the address of the `add` function using the `&` operator.\\\\n\\\\nFinally, we call the `add` function directly and store the result in a variable `result`.\\\\n\\\\nI hope this helps you understand how function pointers work in C! Let me know if you have any other questions.\\\",\\\"To define a function pointer within a struct, you can define a member variable of the struct as a pointer to a function with a specific return type and argument list. Here's an example:\\\\n\\\\n```c\\\\n#include <stdio.h>\\\\n\\\\n\\\\/\\\\/ Define a structure with a function pointer member\\\\ntypedef struct {\\\\n    int (*calculate)(int);\\\\n} Calculator;\\\\n\\\\n\\\\/\\\\/ Function to be pointed to\\\\nint add(int num) {\\\\n    return num + 10;\\\\n}\\\\n\\\\n\\\\/\\\\/ Function to demonstrate using a function pointer within a struct\\\\nvoid useCalculator(Calculator calculator) {\\\\n    int result = calculator.calculate(15);\\\\n    printf(\\\\\\\"Result: %d\\\\\\\\n\\\\\\\", result);\\\\n}\\\\n\\\\nint main() {\\\\n    \\\\/\\\\/ Create a Calculator struct and initialize the function pointer member\\\\n    Calculator calculator = { add };\\\\n\\\\n    \\\\/\\\\/ Call the useCalculator function and pass the calculator struct\\\\n    useCalculator(calculator);\\\\n\\\\n    return 0;\\\\n}\\\\n```\\\\n\\\\nIn this example, we define a `typedef` for our `Calculator` struct, which has a single member `calculate` that is a pointer to a function that takes an integer as an argument and returns an integer.\\\\n\\\\nWe then define a function `add` that adds 10 to the input integer and return the result.\\\\n\\\\nWe define a function `useCalculator` that takes a `Calculator` struct as an argument and invokes the `calculate` member function of the struct with an argument of 15.\\\\n\\\\nIn the `main` function, we create a `Calculator` struct and initialize its `calculate` member with the address of the `add` function. We then call the `useCalculator` function passing the `calculator` struct.\\\\n\\\\nI hope this helps you understand how to define a function pointer within a struct in C! Let me know if you have any other questions.\\\"]\",\n          \"[\\\"In the depths of a dark and mysterious cave, there lived a heavily pregnant cavewoman named Ula. Her thick, matted hair cascaded down her back, and her skin was tanned and rough from years of living in the harsh wilderness. Ula's husband, Kadan, was a skilled hunter and provided for their small tribe. But lately, even his sharpest arrows seemed to fall short, and food had become scarce.\\\\n\\\\nUla felt a deep ache in her bones and her swollen belly. It was the third trimester, and her baby was growing rapidly. She wondered if she would ever be able to give birth in the cramped, narrow cave. Ula's tribe had no midwife, and her only hope was to rely on the wisdom passed down from her own mother.\\\\n\\\\nOne day, while Kadan was out hunting, Ula went into labor. Her screams echoed through the cave, and her tribe gathered around her in a frenzy. Ula's eyes widened as she felt the intense pain of contractions. She was afraid for her baby's life and her own.\\\\n\\\\nThe tribe tried to comfort Ula, but their efforts fell short. Ula's baby was too big, and the birth seemed impossible. Hours passed, and Ula lost all strength. Her tribe watched in horror as Ula's life slipped away. Suddenly, a loud cry broke the silence. Ula's baby had been born, and he was healthy and strong.\\\\n\\\\nThe tribe rejoiced at the sight of the newborn. They named him Kade, after Ula's husband, Kadan. Kade was a symbol of hope and resilience, and his birth brought new life and energy to the tribe.\\\\n\\\\nAs Ula took her last breath, she whispered to Kade, \\\\\\\"May you grow up to be a fierce and proud caveman, just like your father.\\\\\\\" Ula's legacy lived on through her son, and her tribe honored her memory by carving her name into the walls of the cave. The story of Ula, the heavily pregnant cavewoman who defied death to give birth to Kade, became a legend that would be passed down through the generations.\\\"]\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response_b\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 56609,\n        \"samples\": [\n          \"[\\\"Unfortunately, I do not have enough context to provide specific details about \\\\\\\"mobility123\\\\\\\" in New Jersey. Mobility123 is not a term I am familiar with. To provide more useful information, it would be helpful if you could clarify what kind of service, company, or product mobility123 refers to, as well as any additional details about what you would like to know in relation to New Jersey. Please provide some more context and details so that I can try to assist. Let me know if you can share more information!\\\"]\",\n          \"[\\\"Abby, Amber, Amra, Arin, Arna, Ashy, Bari, Bett, Brix, Bryn, Brin, Brynne, Ciri, Cyrn, Ebby, Ebri, Elbr, Elmry, Embr, Raby, Reba\\\"]\",\n          \"[\\\"It's great that your child is considering how to communicate with her friends in a way that doesn't upset them. Here are some tips that may help:\\\\n\\\\n1. Listen actively: Encourage your child to listen to her friends without interrupting or judging them. This means paying attention to what they're saying, asking questions to clarify their thoughts, and acknowledging their feelings.\\\\n2. Use kind words: Remind your child to use kind and respectful language when talking to her friends. This means avoiding hurtful words or tone, and being mindful of how her words may be perceived by others.\\\\n3. Be empathetic: Encourage your child to try to see things from her friends' perspective and to understand their feelings and needs. This means putting herself in their shoes and considering how they might be feeling.\\\\n4. Avoid gossip: Remind your child that gossiping about her friends can be hurtful and damaging to their relationships. Encourage her to avoid talking about her friends behind their backs and to speak to them directly if she has concerns.\\\\n5. Be honest: Remind your child that honesty is always the best policy, even if it's difficult to hear. Encourage her to be open and truthful with her friends, but also to be gentle and considerate in how she delivers her message.\\\\n\\\\nBy following these tips, your child can learn how to communicate effectively with her friends without upsetting them.\\\"]\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"winner_model_a\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"winner_model_b\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"winner_tie\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize the ELO ratings for each model\n",
        "elo_ratings = {}\n",
        "\n",
        "def get_elo(model):\n",
        "    \"\"\"Get the ELO rating for a model, defaulting to 1000.\"\"\"\n",
        "    return elo_ratings.get(model, 1000)\n",
        "\n",
        "def update_elo(winner, loser, k=32, draw=False):\n",
        "    \"\"\"Update ELO ratings for a winner and loser model, or in case of a draw.\"\"\"\n",
        "    winner_elo = get_elo(winner)\n",
        "    loser_elo = get_elo(loser)\n",
        "\n",
        "    # Calculate expected scores\n",
        "    expected_winner = 1 / (1 + 10 ** ((loser_elo - winner_elo) / 400))\n",
        "    expected_loser = 1 / (1 + 10 ** ((winner_elo - loser_elo) / 400))\n",
        "\n",
        "    # Define the score outcome\n",
        "    if draw:\n",
        "        winner_score = loser_score = 0.5\n",
        "    else:\n",
        "        winner_score = 1\n",
        "        loser_score = 0\n",
        "\n",
        "    # Update ratings\n",
        "    elo_ratings[winner] = winner_elo + k * (winner_score - expected_winner)\n",
        "    elo_ratings[loser] = loser_elo + k * (loser_score - expected_loser)\n",
        "\n",
        "# Process each row in the DataFrame to update the ELO ratings\n",
        "for _, row in df.iterrows():\n",
        "    model_a = row['model_a']\n",
        "    model_b = row['model_b']\n",
        "\n",
        "    if row['winner_model_a'] == 1:\n",
        "        update_elo(model_a, model_b)\n",
        "    elif row['winner_model_b'] == 1:\n",
        "        update_elo(model_b, model_a)\n",
        "    elif row['winner_tie'] == 1:\n",
        "        update_elo(model_a, model_b, draw=True)\n",
        "\n",
        "# Convert ELO ratings to a DataFrame for easy viewing\n",
        "elo_df = pd.DataFrame(list(elo_ratings.items()), columns=['Model', 'ELO Rating']).sort_values(by='ELO Rating', ascending=False)\n",
        "elo_df.reset_index(drop=True, inplace=True)\n",
        "\n",
        "# Display the DataFrame using tabulate for a cleaner output\n",
        "elo_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "Bu1V7P5qKcF1",
        "outputId": "3458db06-84dd-43eb-9ffe-11c63df5f31a"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      Model   ELO Rating\n",
              "0        gpt-4-0125-preview  1247.307756\n",
              "1        gpt-3.5-turbo-0613  1174.807565\n",
              "2        gpt-4-1106-preview  1173.268998\n",
              "3                gpt-4-0314  1130.541965\n",
              "4            tulu-2-dpo-70b  1127.553508\n",
              "..                      ...          ...\n",
              "59         oasst-pythia-12b   827.413651\n",
              "60  stablelm-tuned-alpha-7b   809.281896\n",
              "61               chatglm-6b   752.394747\n",
              "62                llama-13b   729.849219\n",
              "63             dolly-v2-12b   720.662150\n",
              "\n",
              "[64 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-122c8a59-2a49-4981-8f56-6e65c6d26bbe\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Model</th>\n",
              "      <th>ELO Rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>gpt-4-0125-preview</td>\n",
              "      <td>1247.307756</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>gpt-3.5-turbo-0613</td>\n",
              "      <td>1174.807565</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>gpt-4-1106-preview</td>\n",
              "      <td>1173.268998</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>gpt-4-0314</td>\n",
              "      <td>1130.541965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>tulu-2-dpo-70b</td>\n",
              "      <td>1127.553508</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>oasst-pythia-12b</td>\n",
              "      <td>827.413651</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>stablelm-tuned-alpha-7b</td>\n",
              "      <td>809.281896</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>chatglm-6b</td>\n",
              "      <td>752.394747</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>llama-13b</td>\n",
              "      <td>729.849219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>dolly-v2-12b</td>\n",
              "      <td>720.662150</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>64 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-122c8a59-2a49-4981-8f56-6e65c6d26bbe')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-122c8a59-2a49-4981-8f56-6e65c6d26bbe button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-122c8a59-2a49-4981-8f56-6e65c6d26bbe');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b3be0c3b-f199-435b-8ddf-d6c452a89004\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b3be0c3b-f199-435b-8ddf-d6c452a89004')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b3be0c3b-f199-435b-8ddf-d6c452a89004 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_d3ed36f4-52c3-4f47-bab1-8542eb303e14\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('elo_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_d3ed36f4-52c3-4f47-bab1-8542eb303e14 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('elo_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "elo_df",
              "summary": "{\n  \"name\": \"elo_df\",\n  \"rows\": 64,\n  \"fields\": [\n    {\n      \"column\": \"Model\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 64,\n        \"samples\": [\n          \"koala-13b\",\n          \"alpaca-13b\",\n          \"gpt-4-0125-preview\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ELO Rating\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 106.64689190394145,\n        \"min\": 720.6621499450347,\n        \"max\": 1247.3077557139534,\n        \"num_unique_values\": 64,\n        \"samples\": [\n          911.052795460804,\n          827.5962283909001,\n          1247.3077557139534\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Grouping into 10 groups\n",
        "n = 10\n",
        "elo_df['Group'] = pd.qcut(elo_df['ELO Rating'], q=n, labels=False)\n",
        "elo_df['Group'] = n-1-elo_df['Group']\n",
        "\n",
        "# Display the grouped DataFrame\n",
        "elo_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "fHIO2S4XLBW_",
        "outputId": "054ab609-649e-4e4d-862c-964be74520d2"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      Model   ELO Rating  Group\n",
              "0        gpt-4-0125-preview  1247.307756      0\n",
              "1        gpt-3.5-turbo-0613  1174.807565      0\n",
              "2        gpt-4-1106-preview  1173.268998      0\n",
              "3                gpt-4-0314  1130.541965      0\n",
              "4            tulu-2-dpo-70b  1127.553508      0\n",
              "..                      ...          ...    ...\n",
              "59         oasst-pythia-12b   827.413651      9\n",
              "60  stablelm-tuned-alpha-7b   809.281896      9\n",
              "61               chatglm-6b   752.394747      9\n",
              "62                llama-13b   729.849219      9\n",
              "63             dolly-v2-12b   720.662150      9\n",
              "\n",
              "[64 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-573d17ac-cb37-40db-8eae-639eb1ee4fee\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Model</th>\n",
              "      <th>ELO Rating</th>\n",
              "      <th>Group</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>gpt-4-0125-preview</td>\n",
              "      <td>1247.307756</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>gpt-3.5-turbo-0613</td>\n",
              "      <td>1174.807565</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>gpt-4-1106-preview</td>\n",
              "      <td>1173.268998</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>gpt-4-0314</td>\n",
              "      <td>1130.541965</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>tulu-2-dpo-70b</td>\n",
              "      <td>1127.553508</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>oasst-pythia-12b</td>\n",
              "      <td>827.413651</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>stablelm-tuned-alpha-7b</td>\n",
              "      <td>809.281896</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>chatglm-6b</td>\n",
              "      <td>752.394747</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>llama-13b</td>\n",
              "      <td>729.849219</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>dolly-v2-12b</td>\n",
              "      <td>720.662150</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>64 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-573d17ac-cb37-40db-8eae-639eb1ee4fee')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-573d17ac-cb37-40db-8eae-639eb1ee4fee button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-573d17ac-cb37-40db-8eae-639eb1ee4fee');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-788e4802-1625-4906-aa77-ccb0953268ff\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-788e4802-1625-4906-aa77-ccb0953268ff')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-788e4802-1625-4906-aa77-ccb0953268ff button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_7f725154-2c39-4150-95fe-5a19148d2785\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('elo_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_7f725154-2c39-4150-95fe-5a19148d2785 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('elo_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "elo_df",
              "summary": "{\n  \"name\": \"elo_df\",\n  \"rows\": 64,\n  \"fields\": [\n    {\n      \"column\": \"Model\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 64,\n        \"samples\": [\n          \"koala-13b\",\n          \"alpaca-13b\",\n          \"gpt-4-0125-preview\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ELO Rating\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 106.64689190394145,\n        \"min\": 720.6621499450347,\n        \"max\": 1247.3077557139534,\n        \"num_unique_values\": 64,\n        \"samples\": [\n          911.052795460804,\n          827.5962283909001,\n          1247.3077557139534\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Group\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2,\n        \"min\": 0,\n        \"max\": 9,\n        \"num_unique_values\": 10,\n        \"samples\": [\n          8,\n          1,\n          5\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "strong_models = elo_df[elo_df.Group <=1]['Model'].unique()\n",
        "strong_models"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aUjXBQ3eMiKj",
        "outputId": "d961cc0a-419d-4308-8c6e-3b745c7cc4bc"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['gpt-4-0125-preview', 'gpt-3.5-turbo-0613', 'gpt-4-1106-preview',\n",
              "       'gpt-4-0314', 'tulu-2-dpo-70b', 'llama-2-70b-chat',\n",
              "       'qwen1.5-72b-chat', 'nous-hermes-2-mixtral-8x7b-dpo', 'gemini-pro',\n",
              "       'mixtral-8x7b-instruct-v0.1', 'llama-2-7b-chat', 'mistral-medium',\n",
              "       'wizardlm-70b'], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "weak_models = elo_df[(elo_df.Group>1) & (elo_df.Group<=3)]['Model'].unique()\n",
        "weak_models"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Okb_P5U1N0X0",
        "outputId": "064e3c4c-4869-4ebe-ad4d-aade7e584731"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['gpt-3.5-turbo-0314', 'gpt-3.5-turbo-0125',\n",
              "       'openhermes-2.5-mistral-7b', 'pplx-70b-online', 'claude-2.0',\n",
              "       'gpt-4-0613', 'gemini-pro-dev-api', 'claude-2.1', 'claude-1',\n",
              "       'llama2-70b-steerlm-chat', 'deepseek-llm-67b-chat',\n",
              "       'solar-10.7b-instruct-v1.0', 'wizardlm-13b'], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Filter rows where there's a match between a strong and a weak model\n",
        "matches = df[((df['model_a'].isin(strong_models)) & (df['model_b'].isin(weak_models))) |\n",
        "             ((df['model_a'].isin(weak_models)) & (df['model_b'].isin(strong_models)))].copy()\n",
        "\n",
        "# Reconstruct DataFrame to show whether strong or weak model won\n",
        "def identify_winner(row):\n",
        "    if row['model_a'] in strong_models and row['winner_model_a'] == 1:\n",
        "        return 'strong'\n",
        "    elif row['model_b'] in strong_models and row['winner_model_b'] == 1:\n",
        "        return 'strong'\n",
        "    elif row['model_a'] in weak_models and row['winner_model_a'] == 1:\n",
        "        return 'weak'\n",
        "    elif row['model_b'] in weak_models and row['winner_model_b'] == 1:\n",
        "        return 'weak'\n",
        "    else:\n",
        "        return 'tie'\n",
        "\n",
        "# Apply the identify_winner function to create a new column\n",
        "matches.loc[:, 'Winner'] = matches.apply(identify_winner, axis=1)  # Use .loc to set the new column\n",
        "\n",
        "# Select and rearrange columns for clarity\n",
        "result_df = matches[['Winner', 'prompt']]\n",
        "result_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "UJLsw0_7P-6K",
        "outputId": "3d135df6-e668-43b5-d98d-207e44fe8de2"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       Winner                                             prompt\n",
              "0      strong  [\"Is it morally right to try to have a certain...\n",
              "7        weak  [\"\\\"Bacteria is life on Mars but a heartbeat i...\n",
              "22        tie  [\"Can you explain what the Cypher Query Langua...\n",
              "24        tie  [\"Can you write a story about a lifeguard name...\n",
              "26        tie  [\"could you tell a funny and whimsical story a...\n",
              "...       ...                                                ...\n",
              "57465  strong  [\"please describe how the following two statem...\n",
              "57466  strong  [\"this is a simulation. 37 years old, white ma...\n",
              "57467  strong  [\"can you provide a modular python code that w...\n",
              "57470  strong  [\"I want you to act like {character} from {ser...\n",
              "57476    weak  [\"three kids eat three apples in three days, h...\n",
              "\n",
              "[11115 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0daa775e-772a-4209-b28e-1390f60f6383\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Winner</th>\n",
              "      <th>prompt</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"Is it morally right to try to have a certain...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>weak</td>\n",
              "      <td>[\"\\\"Bacteria is life on Mars but a heartbeat i...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>tie</td>\n",
              "      <td>[\"Can you explain what the Cypher Query Langua...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>tie</td>\n",
              "      <td>[\"Can you write a story about a lifeguard name...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>tie</td>\n",
              "      <td>[\"could you tell a funny and whimsical story a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57465</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"please describe how the following two statem...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57466</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"this is a simulation. 37 years old, white ma...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57467</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"can you provide a modular python code that w...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57470</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"I want you to act like {character} from {ser...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57476</th>\n",
              "      <td>weak</td>\n",
              "      <td>[\"three kids eat three apples in three days, h...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>11115 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0daa775e-772a-4209-b28e-1390f60f6383')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-0daa775e-772a-4209-b28e-1390f60f6383 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-0daa775e-772a-4209-b28e-1390f60f6383');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-1c6a9228-292b-4d76-a1d2-b780231e8851\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1c6a9228-292b-4d76-a1d2-b780231e8851')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-1c6a9228-292b-4d76-a1d2-b780231e8851 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_f8d4b152-2426-4cd5-82e6-563aa6729576\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('result_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_f8d4b152-2426-4cd5-82e6-563aa6729576 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('result_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "result_df",
              "summary": "{\n  \"name\": \"result_df\",\n  \"rows\": 11115,\n  \"fields\": [\n    {\n      \"column\": \"Winner\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"strong\",\n          \"weak\",\n          \"tie\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"prompt\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10627,\n        \"samples\": [\n          \"[\\\"Write a poem\\\"]\",\n          \"[\\\"generate a Java webserver to provide weather forecast to user \\\"]\",\n          \"[\\\"code a python ai to play ping pong automatically, the ai will auto learn with the user, the user movement is mouse, add a score handle too.\\\",\\\"continue\\\"]\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_df.Winner.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 210
        },
        "id": "QCj4pnbRTOI3",
        "outputId": "8e17b8e9-c7d1-4cf5-ef71-fb3e8743e145"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Winner\n",
              "strong    4606\n",
              "tie       3335\n",
              "weak      3174\n",
              "Name: count, dtype: int64"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Winner</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>strong</th>\n",
              "      <td>4606</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tie</th>\n",
              "      <td>3335</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>weak</th>\n",
              "      <td>3174</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div><br><label><b>dtype:</b> int64</label>"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Note\n",
        "RouteLLM augments this data with golden lable data and LLM as judge data. We won't do that here."
      ],
      "metadata": {
        "id": "0rzPam1VUXKQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Routing with BT model\n",
        "\n",
        "Using a Bradley-Terry model with embeddings to decide which model class to route to"
      ],
      "metadata": {
        "id": "PKsAz-dRX3Dj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install openai\n",
        "!pip install langchain-community\n",
        "!pip install tiktoken"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WbFDluu4gsE8",
        "outputId": "e2c57939-da39-45d9-fcfa-ddcb9a168488"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: openai in /usr/local/lib/python3.10/dist-packages (1.54.3)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.9.0)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.27.2)\n",
            "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.7.0)\n",
            "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (2.9.2)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai) (1.3.1)\n",
            "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.6)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.11 in /usr/local/lib/python3.10/dist-packages (from openai) (4.12.2)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (3.10)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (1.2.2)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2024.8.30)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (1.0.6)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai) (2.23.4)\n",
            "Collecting langchain-community\n",
            "  Downloading langchain_community-0.3.7-py3-none-any.whl.metadata (2.9 kB)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (6.0.2)\n",
            "Collecting SQLAlchemy<2.0.36,>=1.4 (from langchain-community)\n",
            "  Downloading SQLAlchemy-2.0.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.6 kB)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (3.10.10)\n",
            "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain-community)\n",
            "  Downloading dataclasses_json-0.6.7-py3-none-any.whl.metadata (25 kB)\n",
            "Collecting httpx-sse<0.5.0,>=0.4.0 (from langchain-community)\n",
            "  Downloading httpx_sse-0.4.0-py3-none-any.whl.metadata (9.0 kB)\n",
            "Requirement already satisfied: langchain<0.4.0,>=0.3.7 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (0.3.7)\n",
            "Collecting langchain-core<0.4.0,>=0.3.17 (from langchain-community)\n",
            "  Downloading langchain_core-0.3.18-py3-none-any.whl.metadata (6.3 kB)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.125 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (0.1.142)\n",
            "Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (1.26.4)\n",
            "Collecting pydantic-settings<3.0.0,>=2.4.0 (from langchain-community)\n",
            "  Downloading pydantic_settings-2.6.1-py3-none-any.whl.metadata (3.5 kB)\n",
            "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (2.32.3)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-community) (9.0.0)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (2.4.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (24.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.5.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (6.1.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.17.1)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (4.0.3)\n",
            "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community)\n",
            "  Downloading marshmallow-3.23.1-py3-none-any.whl.metadata (7.5 kB)\n",
            "Collecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain-community)\n",
            "  Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)\n",
            "Requirement already satisfied: langchain-text-splitters<0.4.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from langchain<0.4.0,>=0.3.7->langchain-community) (0.3.2)\n",
            "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in /usr/local/lib/python3.10/dist-packages (from langchain<0.4.0,>=0.3.7->langchain-community) (2.9.2)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.4.0,>=0.3.17->langchain-community) (1.33)\n",
            "Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.4.0,>=0.3.17->langchain-community) (24.2)\n",
            "Requirement already satisfied: typing-extensions>=4.7 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.4.0,>=0.3.17->langchain-community) (4.12.2)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.125->langchain-community) (0.27.2)\n",
            "Requirement already satisfied: orjson<4.0.0,>=3.9.14 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.125->langchain-community) (3.10.11)\n",
            "Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.2.0,>=0.1.125->langchain-community) (1.0.0)\n",
            "Collecting python-dotenv>=0.21.0 (from pydantic-settings<3.0.0,>=2.4.0->langchain-community)\n",
            "  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-community) (2024.8.30)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<2.0.36,>=1.4->langchain-community) (3.1.1)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->langsmith<0.2.0,>=0.1.125->langchain-community) (3.7.1)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->langsmith<0.2.0,>=0.1.125->langchain-community) (1.0.6)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->langsmith<0.2.0,>=0.1.125->langchain-community) (1.3.1)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->langsmith<0.2.0,>=0.1.125->langchain-community) (0.14.0)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.17->langchain-community) (3.0.0)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain<0.4.0,>=0.3.7->langchain-community) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain<0.4.0,>=0.3.7->langchain-community) (2.23.4)\n",
            "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community)\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp<4.0.0,>=3.8.3->langchain-community) (0.2.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx<1,>=0.23.0->langsmith<0.2.0,>=0.1.125->langchain-community) (1.2.2)\n",
            "Downloading langchain_community-0.3.7-py3-none-any.whl (2.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB)\n",
            "Downloading httpx_sse-0.4.0-py3-none-any.whl (7.8 kB)\n",
            "Downloading langchain_core-0.3.18-py3-none-any.whl (409 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m409.3/409.3 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pydantic_settings-2.6.1-py3-none-any.whl (28 kB)\n",
            "Downloading SQLAlchemy-2.0.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m52.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading marshmallow-3.23.1-py3-none-any.whl (49 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.5/49.5 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
            "Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
            "Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "Installing collected packages: SQLAlchemy, python-dotenv, mypy-extensions, marshmallow, httpx-sse, typing-inspect, pydantic-settings, dataclasses-json, langchain-core, langchain-community\n",
            "  Attempting uninstall: SQLAlchemy\n",
            "    Found existing installation: SQLAlchemy 2.0.36\n",
            "    Uninstalling SQLAlchemy-2.0.36:\n",
            "      Successfully uninstalled SQLAlchemy-2.0.36\n",
            "  Attempting uninstall: langchain-core\n",
            "    Found existing installation: langchain-core 0.3.15\n",
            "    Uninstalling langchain-core-0.3.15:\n",
            "      Successfully uninstalled langchain-core-0.3.15\n",
            "Successfully installed SQLAlchemy-2.0.35 dataclasses-json-0.6.7 httpx-sse-0.4.0 langchain-community-0.3.7 langchain-core-0.3.18 marshmallow-3.23.1 mypy-extensions-1.0.0 pydantic-settings-2.6.1 python-dotenv-1.0.1 typing-inspect-0.9.0\n",
            "Collecting tiktoken\n",
            "  Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2024.9.11)\n",
            "Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2.32.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2024.8.30)\n",
            "Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: tiktoken\n",
            "Successfully installed tiktoken-0.8.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Embedding Data"
      ],
      "metadata": {
        "id": "tiYpgDxeg9Av"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import userdata\n",
        "import openai\n",
        "import os\n",
        "\n",
        "api_key = userdata.get('OpenAIAPIKey')\n",
        "os.environ['OPENAI_API_KEY'] = api_key"
      ],
      "metadata": {
        "id": "m6UlVrF3hR4t"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from langchain.embeddings import OpenAIEmbeddings\n",
        "\n",
        "# Initialize the LangChain OpenAI embeddings model\n",
        "embeddings_model = OpenAIEmbeddings(model=\"text-embedding-ada-002\")  # Use the small model\n",
        "\n",
        "# Get batch embeddings for all prompts in result_df\n",
        "prompts = result_df['prompt'].tolist()\n",
        "\n",
        "#sanatizing\n",
        "prompts = [p.replace('<|endoftext|>', '') for p in prompts]\n",
        "\n",
        "result_df['prompt_embedding'] = embeddings_model.embed_documents(prompts)"
      ],
      "metadata": {
        "id": "uYS37eP5gtG4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1d6153f1-6d49-4611-90fb-abcda3205219"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-11-bcf021f30eac>:6: LangChainDeprecationWarning: The class `OpenAIEmbeddings` was deprecated in LangChain 0.0.9 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-openai package and should be used instead. To use it run `pip install -U :class:`~langchain-openai` and import as `from :class:`~langchain_openai import OpenAIEmbeddings``.\n",
            "  embeddings_model = OpenAIEmbeddings(model=\"text-embedding-ada-002\")  # Use the small model\n",
            "<ipython-input-11-bcf021f30eac>:14: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  result_df['prompt_embedding'] = embeddings_model.embed_documents(prompts)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "jUr4AAaETzN0",
        "outputId": "7706fd70-8508-4f2d-b27f-1b1f82f657bc"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       Winner                                             prompt  \\\n",
              "0      strong  [\"Is it morally right to try to have a certain...   \n",
              "7        weak  [\"\\\"Bacteria is life on Mars but a heartbeat i...   \n",
              "22        tie  [\"Can you explain what the Cypher Query Langua...   \n",
              "24        tie  [\"Can you write a story about a lifeguard name...   \n",
              "26        tie  [\"could you tell a funny and whimsical story a...   \n",
              "...       ...                                                ...   \n",
              "57465  strong  [\"please describe how the following two statem...   \n",
              "57466  strong  [\"this is a simulation. 37 years old, white ma...   \n",
              "57467  strong  [\"can you provide a modular python code that w...   \n",
              "57470  strong  [\"I want you to act like {character} from {ser...   \n",
              "57476    weak  [\"three kids eat three apples in three days, h...   \n",
              "\n",
              "                                        prompt_embedding  \n",
              "0      [0.016110674600082704, -0.02285776545015528, 0...  \n",
              "7      [-0.010212965153490029, -0.002264541306372864,...  \n",
              "22     [0.0214463146508967, 0.008269389523151266, 0.0...  \n",
              "24     [2.6210532231287662e-05, -0.000634177259772841...  \n",
              "26     [0.015002578925881449, -0.008475396619294216, ...  \n",
              "...                                                  ...  \n",
              "57465  [0.0053320008910354095, 0.008999157480533974, ...  \n",
              "57466  [-0.02408318648581002, 0.0017563447287897128, ...  \n",
              "57467  [0.016568595443204243, 0.021091258025641487, -...  \n",
              "57470  [-0.01874203756635266, 0.012563647453844166, 0...  \n",
              "57476  [0.02329920790775893, 0.0018384531137852623, 0...  \n",
              "\n",
              "[11115 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9355d5b7-c3d1-4172-b1ee-0158fe4a5d85\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Winner</th>\n",
              "      <th>prompt</th>\n",
              "      <th>prompt_embedding</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"Is it morally right to try to have a certain...</td>\n",
              "      <td>[0.016110674600082704, -0.02285776545015528, 0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>weak</td>\n",
              "      <td>[\"\\\"Bacteria is life on Mars but a heartbeat i...</td>\n",
              "      <td>[-0.010212965153490029, -0.002264541306372864,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>tie</td>\n",
              "      <td>[\"Can you explain what the Cypher Query Langua...</td>\n",
              "      <td>[0.0214463146508967, 0.008269389523151266, 0.0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>tie</td>\n",
              "      <td>[\"Can you write a story about a lifeguard name...</td>\n",
              "      <td>[2.6210532231287662e-05, -0.000634177259772841...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>tie</td>\n",
              "      <td>[\"could you tell a funny and whimsical story a...</td>\n",
              "      <td>[0.015002578925881449, -0.008475396619294216, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57465</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"please describe how the following two statem...</td>\n",
              "      <td>[0.0053320008910354095, 0.008999157480533974, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57466</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"this is a simulation. 37 years old, white ma...</td>\n",
              "      <td>[-0.02408318648581002, 0.0017563447287897128, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57467</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"can you provide a modular python code that w...</td>\n",
              "      <td>[0.016568595443204243, 0.021091258025641487, -...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57470</th>\n",
              "      <td>strong</td>\n",
              "      <td>[\"I want you to act like {character} from {ser...</td>\n",
              "      <td>[-0.01874203756635266, 0.012563647453844166, 0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57476</th>\n",
              "      <td>weak</td>\n",
              "      <td>[\"three kids eat three apples in three days, h...</td>\n",
              "      <td>[0.02329920790775893, 0.0018384531137852623, 0...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>11115 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9355d5b7-c3d1-4172-b1ee-0158fe4a5d85')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9355d5b7-c3d1-4172-b1ee-0158fe4a5d85 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9355d5b7-c3d1-4172-b1ee-0158fe4a5d85');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2a4ed16f-ef7f-4a6a-b373-98cdc8db9a1b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2a4ed16f-ef7f-4a6a-b373-98cdc8db9a1b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2a4ed16f-ef7f-4a6a-b373-98cdc8db9a1b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_aef55772-f0e5-43f9-931f-e332770ec7e3\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('result_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_aef55772-f0e5-43f9-931f-e332770ec7e3 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('result_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "result_df",
              "summary": "{\n  \"name\": \"result_df\",\n  \"rows\": 11115,\n  \"fields\": [\n    {\n      \"column\": \"Winner\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"strong\",\n          \"weak\",\n          \"tie\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"prompt\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10627,\n        \"samples\": [\n          \"[\\\"Write a poem\\\"]\",\n          \"[\\\"generate a Java webserver to provide weather forecast to user \\\"]\",\n          \"[\\\"code a python ai to play ping pong automatically, the ai will auto learn with the user, the user movement is mouse, add a score handle too.\\\",\\\"continue\\\"]\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"prompt_embedding\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Creating Similarity Based Routing"
      ],
      "metadata": {
        "id": "TqP96J039-V5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def calculate_weight(new_query_embedding, training_query_embeddings, gamma=10):\n",
        "    \"\"\"Returns a weight for each training query embedding via normalized cosine similarity.\n",
        "    Parameters:\n",
        "    - new_query_embedding: numpy array of shape [embedding_dim], the embedding of the new query.\n",
        "    - training_query_embeddings: numpy array of shape [N, embedding_dim], embeddings of training queries.\n",
        "    - gamma: float, scaling factor for the weight calculation.\n",
        "    Returns:\n",
        "    - weights: numpy array of shape [N], weights for each training query embedding.\n",
        "    \"\"\"\n",
        "\n",
        "    #calculating cosine similarity of the new query vs all training queries\n",
        "    dot = np.dot(training_query_embeddings, new_query_embedding)\n",
        "    norms_training = np.linalg.norm(training_query_embeddings, axis=1, keepdims=True)\n",
        "    norm_new = np.linalg.norm(new_query_embedding)\n",
        "    cosine_similarities = (dot/(norms_training.T*norm_new))[0]\n",
        "\n",
        "    #Normalizing cosine similarities by max value\n",
        "    normalized_similarities = cosine_similarities/max(cosine_similarities)\n",
        "\n",
        "    #calculating weight\n",
        "    return gamma ** (1+normalized_similarities)\n",
        "\n",
        "# Testing with sample data\n",
        "# Create a random embedding for the new query\n",
        "np.random.seed(0)\n",
        "new_query_embedding = np.random.rand(10)\n",
        "\n",
        "# Create random embeddings for training queries\n",
        "training_query_embeddings = np.random.rand(10, 10)\n",
        "\n",
        "# Calculate weights\n",
        "weights = calculate_weight(new_query_embedding, training_query_embeddings, gamma=10)\n",
        "weights"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XvcTslyu9g1r",
        "outputId": "0bbc543d-40ca-44d5-f471-e4628562f9fc"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 82.70779993,  95.36947674, 100.        ,  72.57732008,\n",
              "        72.75448222,  65.91411544,  60.36960324,  65.01629365,\n",
              "        71.35973296,  74.87936352])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Minimizing Weighted Binary Cross Entropy To Find Parameters"
      ],
      "metadata": {
        "id": "jZ1EU_6fG_S1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def compute_coefficients(weights, labels, learning_rate=0.01, epochs=20):\n",
        "    \"\"\"Computes the Bradley-Terry Coefficients based on the weights and labels for each query.\n",
        "\n",
        "    Parameters:\n",
        "    - weights: numpy array of shape [N], weights for each training query embedding.\n",
        "    - labels: numpy array of shape [N], binary labels (0 or 1) for each training example.\n",
        "    - learning_rate: float, learning rate for gradient descent.\n",
        "    - epochs: int, number of iterations for gradient descent.\n",
        "\n",
        "    Returns:\n",
        "    - xi: numpy array of shape [2], optimized BT coefficients for small and large models\n",
        "    \"\"\"\n",
        "\n",
        "    def binary_cross_entropy_loss(label, probability):\n",
        "        \"\"\"Calculate binary cross-entropy loss for a single example with probability clipping to avoid log(0).\"\"\"\n",
        "        epsilon = 1e-10  # Small value to prevent log(0)\n",
        "        probability = np.clip(probability, epsilon, 1 - epsilon)\n",
        "        return -(label * np.log(probability) + (1 - label) * np.log(1 - probability))\n",
        "\n",
        "    # Initialize coefficients (xi) randomly, one for each class\n",
        "    # In the RouteLLM paper they learn one coefficient for 10 parititions,\n",
        "    # But I found the details to be confusing. I'm computing coefficients for\n",
        "    # small models and large models as we defined previously\n",
        "\n",
        "    #the labels represent if large is necessary (1) or not (0), so index 0\n",
        "    #of this should represent the coefficient of the large model not being\n",
        "    #necessary, and index 1 represents the coefficient large model being\n",
        "    #necessary. These coefficients will be optimized to bias one vs the other.\n",
        "\n",
        "    # note: The BT model is optimized for *every query*, meaning\n",
        "    # these will be optimized based on the similarities weights\n",
        "    # for an individual case.\n",
        "    xi = np.random.rand(2)\n",
        "\n",
        "    #if there's a strong bias towards one model or the other throughout the data,\n",
        "    # the BT model will learn that bias heavily, and thus won't learn routing nuance.\n",
        "    #As a result, we're going to balance against that bias by biasing the gradient\n",
        "    #of the loss when the label indicates a large model. If the large model is\n",
        "    #called more often in the training, then this value will be <1. If less often,\n",
        "    #this value will be >1.\n",
        "    large_model_bias = 1/(labels.sum()/len(labels))\n",
        "\n",
        "\n",
        "    for epoch in range(epochs):\n",
        "        total_loss = 0\n",
        "        gradients = np.zeros_like(xi)\n",
        "\n",
        "        for i, (weight, label) in enumerate(zip(weights, labels)):\n",
        "            # Calculate the probability using the Bradley-Terry model\n",
        "            # this essentially asks the bradley terry model, based on the two\n",
        "            # coefficients, how likely it is that a large model is necessary (1)\n",
        "            delta = np.clip(xi[1] - xi[0], -500, 500)  # Clipping to prevent overflow in exp\n",
        "            probability = 1 / (1 + np.exp(-delta)) #probability of a large model being required\n",
        "\n",
        "            # Calculate the cross-entropy loss\n",
        "            loss = binary_cross_entropy_loss(label, probability)\n",
        "            weighted_loss = weight * loss\n",
        "            total_loss += weighted_loss\n",
        "\n",
        "            #Scaling loss if the model is large, according to bias\n",
        "            if label:\n",
        "                loss = large_model_bias*loss\n",
        "\n",
        "            # Calculating gradients\n",
        "            # Here I'm seperatly defining the direction to push the gradients,\n",
        "            # based on the label, and the magnitude of that push based on the\n",
        "            # weight and loss.\n",
        "\n",
        "            # direction is based on if the prediction should be towards large\n",
        "            # +1 for towards large, -1 for towards small\n",
        "            direction = (label*2-1)\n",
        "\n",
        "            gradient_large = direction*(weight * loss)\n",
        "            graident_small =(-1*direction)*(weight * loss)\n",
        "\n",
        "            # xi[0] = small model, and xi[1] = large model\n",
        "            gradients = np.array([graident_small, gradient_large])\n",
        "\n",
        "        # it made more sense to me to represent the chang in terms\n",
        "        # of what should be happening, rather than minimizing loss,\n",
        "        # so here we're adding the gradients rather than subtracting them\n",
        "        xi += learning_rate * gradients\n",
        "\n",
        "        # Optional: Print loss for monitoring\n",
        "        if epoch % 5 == 0:\n",
        "            print(f\"Epoch {epoch}, Total Loss: {total_loss}\")\n",
        "\n",
        "    return xi\n",
        "\n",
        "if False:\n",
        "    # Testing the functions with sample data\n",
        "    # Create random embeddings for new and training queries\n",
        "    np.random.seed(10)\n",
        "    n_classes = 2\n",
        "    n_examples = 10\n",
        "    embedding_dim = 20\n",
        "\n",
        "    new_query_embedding = np.random.rand(embedding_dim)\n",
        "    training_query_embeddings = np.random.rand(n_examples, embedding_dim)\n",
        "\n",
        "    # Calculate weights\n",
        "    weights = calculate_weight(new_query_embedding, training_query_embeddings, gamma=10)\n",
        "\n",
        "    # Generate random binary, corresponding to if a large is necessary (1) or not (0)\n",
        "    labels = np.random.randint(0, n_classes, size=n_examples)\n",
        "\n",
        "    # Compute BT coefficients\n",
        "    coefficients = compute_coefficients(weights, labels, learning_rate=0.01, epochs=10)\n",
        "    print(\"Computed BT coefficients:\", coefficients)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gw_GIidOHbD8",
        "outputId": "07ed10ca-1f20-4938-afed-7eb099d414f4"
      },
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0, Total Loss: 652.3493646160815\n",
            "Epoch 5, Total Loss: 1380.8699159572539\n",
            "Computed BT coefficients: [ 1.6421522  -1.21424972]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "\n",
        "#reformatting the training data\n",
        "bt_data = result_df[['prompt_embedding', 'Winner']].reset_index(drop=True)\n",
        "bt_data.loc[:, 'Winner'] = bt_data['Winner'].map({'strong':1, 'weak':0, 'tie':0})\n",
        "\n",
        "labels = bt_data['Winner'].values\n",
        "embeddings = np.array(list(bt_data['prompt_embedding'].values))\n",
        "\n",
        "#defining a function for computing the bradley terry model\n",
        "#and routing accordingly\n",
        "def route_with_BT_model(query, embeddings, threshold=0.5):\n",
        "    \"\"\"This function essentially does _ things:\n",
        "    1. it embeds the query\n",
        "    2. it compares the embedded query to the embedded training data,\n",
        "    and uses those comparisons to construct a bradley terry model\n",
        "    3. it uses the bradley terry model (which is essentially just two\n",
        "    coefficients) to make a prediction of if a large or small model is\n",
        "    appropriate\n",
        "    4. it compares that probability with a probability threshold, allowing\n",
        "    us to control the implicit cost/performance tradeoff of using small and\n",
        "    large llms\n",
        "\n",
        "    the threshold essentially says how confident we need to be that a small\n",
        "    model is sufficient in order to route to it. 0.5 would lean on the BT\n",
        "    model, <0.5 would be cost saving, and >0.5 would bias for performance.\n",
        "    \"\"\"\n",
        "\n",
        "    #embedding query\n",
        "    query_embedding = np.array(embeddings_model.embed_documents(query)[0])\n",
        "\n",
        "    #calculating weights, based on the training datas similarity\n",
        "    #to the query. gamma=10 is recommended\n",
        "    weights = calculate_weight(query_embedding, embeddings, gamma=10)\n",
        "\n",
        "    #optimizing a Bradley-Terry model, based on the weights\n",
        "    coefficients = compute_coefficients(weights, labels, learning_rate=0.01, epochs=10)\n",
        "\n",
        "    print('coefficients:')\n",
        "    print(coefficients)\n",
        "\n",
        "    #computing probability based on the coefficients\n",
        "    #the way the data is set up, a value of 1 is routing to large models,\n",
        "    #and a value of 0 routes to small models.\n",
        "    prob_large = 1/(1+np.exp(coefficients[0]-coefficients[1]))\n",
        "\n",
        "    return prob_large\n",
        "\n",
        "indexes = [0,1,2,3,4,5,6,7]\n",
        "\n",
        "for i in indexes:\n",
        "    print('======================')\n",
        "    s = result_df.iloc[i]\n",
        "    print(s['Winner'])\n",
        "    prom_large = route_with_BT_model(s['prompt'], embeddings[:10,:])\n",
        "    print(f'predicted {prom_large}, should have predicted {round(s[\"winner_label\"])}')\n",
        ""
      ],
      "metadata": {
        "id": "kdStaKZ8yGhM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb1e1d8b-0ca3-4de5-b08a-8ceababa9830"
      },
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "======================\n",
            "strong\n",
            "Epoch 0, Total Loss: 529.4057273970719\n",
            "Epoch 5, Total Loss: 552.0445242842533\n",
            "coefficients:\n",
            "[ 2.05543539 -1.08473592]\n",
            "predicted 0.041480307498793945, should have predicted 1\n",
            "======================\n",
            "weak\n",
            "Epoch 0, Total Loss: 721.9206215513445\n",
            "Epoch 5, Total Loss: 551.071423693287\n",
            "coefficients:\n",
            "[ 1.85897145 -1.2779113 ]\n",
            "predicted 0.041611256726288355, should have predicted 0\n",
            "======================\n",
            "tie\n",
            "Epoch 0, Total Loss: 555.7080391663139\n",
            "Epoch 5, Total Loss: 550.8295013753294\n",
            "coefficients:\n",
            "[ 2.39561869 -0.74023182]\n",
            "predicted 0.041652441805390676, should have predicted 0\n",
            "======================\n",
            "tie\n",
            "Epoch 0, Total Loss: 586.6766197578112\n",
            "Epoch 5, Total Loss: 550.2530059600223\n",
            "coefficients:\n",
            "[ 1.68666753 -1.44667457]\n",
            "predicted 0.041752686438525644, should have predicted 0\n",
            "======================\n",
            "tie\n",
            "Epoch 0, Total Loss: 598.8569477826356\n",
            "Epoch 5, Total Loss: 550.1419229014549\n",
            "coefficients:\n",
            "[ 2.47207586 -0.66078291]\n",
            "predicted 0.041772028679176905, should have predicted 0\n",
            "======================\n",
            "tie\n",
            "Epoch 0, Total Loss: 659.6426155994552\n",
            "Epoch 5, Total Loss: 550.3576243877756\n",
            "coefficients:\n",
            "[ 2.46251601 -0.67071231]\n",
            "predicted 0.04175723929619793, should have predicted 0\n",
            "======================\n",
            "weak\n",
            "Epoch 0, Total Loss: 720.694349553873\n",
            "Epoch 5, Total Loss: 551.2620481565672\n",
            "coefficients:\n",
            "[ 2.33331019 -0.8034579 ]\n",
            "predicted 0.041615829909134976, should have predicted 0\n",
            "======================\n",
            "strong\n",
            "Epoch 0, Total Loss: 613.826205244236\n",
            "Epoch 5, Total Loss: 550.0738145720563\n",
            "coefficients:\n",
            "[ 1.69664577 -1.43591665]\n",
            "predicted 0.0417838922144574, should have predicted 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "math.ceil(result_df.iloc[2]['winner_label'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NzrQpk_rfr8E",
        "outputId": "b20d482e-479d-45b1-e394-a2a29697cac9"
      },
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "series = result_df.iloc[2]\n",
        "print(series)\n",
        "print(series['prompt'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LFIxO8JvL9AA",
        "outputId": "e3588d73-29a5-4ff8-9977-cd2e1e9df2a8"
      },
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Winner                                                            tie\n",
            "prompt              [\"Can you explain what the Cypher Query Langua...\n",
            "prompt_embedding    [0.0214463146508967, 0.008269389523151266, 0.0...\n",
            "winner_label                                                      0.5\n",
            "Name: 22, dtype: object\n",
            "[\"Can you explain what the Cypher Query Language is?\"]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Underlying Theory\n",
        "So I'm having a hard time getting the BT model to converge. Maybe I'm messing something up, but also I'm curious how well this type of data really clusters. What is the relative distance of each embedding to all other embeddings, and how well do those relative differences neatly seperate for stong and weak models?\n",
        "\n",
        "If this space is too noisy, I feel like there's no way such a simple model would do a good job.\n"
      ],
      "metadata": {
        "id": "NFv5e4u7hDOF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Exploring the underlying distance distributions of weak and strong queries\n",
        "def explore_closness_of_index(result_df, index):\n",
        "    \"\"\"finds how close index is to all others\n",
        "    and returns key metrics\n",
        "    \"\"\"\n",
        "\n",
        "    #getting the partitions being analyzed\n",
        "    df = result_df.reset_index(drop=True)\n",
        "    item = df.loc[index]\n",
        "    df_other = df.drop(index, axis=0)\n",
        "\n",
        "    #getting the embeddings\n",
        "    embedding = np.array(item['prompt_embedding'])\n",
        "    other_embeddings = np.array(list(df_other['prompt_embedding'].values))\n",
        "\n",
        "    #getting labels of what model should be chosen\n",
        "    item_label = round(item['winner_label'])\n",
        "    other_labels = round(df_other['winner_label'])\n",
        "\n",
        "    #weight, as previously defined, is essentially just cosine similarity normalized,\n",
        "    #so I can use that to conveniently calculate closeness\n",
        "    weights = calculate_weight(embedding, embeddings, gamma=1.1)\n",
        "\n",
        "    # Separate closeness values based on whether they match the target label\n",
        "    same_label_closenesses = [c for c, l in zip(weights, other_labels) if l == item_label]\n",
        "    different_label_closenesses = [c for c, l in zip(weights, other_labels) if l != item_label]\n",
        "\n",
        "    # Plot the overlayed histogram\n",
        "    plt.hist(same_label_closenesses, bins=15, alpha=0.5, label='Same Label', edgecolor='black', density=True)\n",
        "    plt.hist(different_label_closenesses, bins=15, alpha=0.5, label='Different Label', edgecolor='black', density=True)\n",
        "\n",
        "    # Add labels and legend\n",
        "    plt.xlabel('Closeness')\n",
        "    plt.ylabel('Frequency')\n",
        "    plt.title('Overlayed Histogram of Closenesses by Label')\n",
        "    plt.legend()\n",
        "\n",
        "    # Display the plot\n",
        "    plt.show()\n",
        "\n",
        "    return weights\n",
        "\n",
        "explore_closness_of_index(result_df, 4)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 508
        },
        "id": "FeJpY3_jhBkE",
        "outputId": "625f408f-308c-431b-f9d6-441b93aa39ef"
      },
      "execution_count": 130,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABcqElEQVR4nO3deVhUZf8G8HvYZoCBGfZFUVBxJ0xLwzVXNDW3cskSzdRetVxSy3LJJcndXFEzXMosfclWNTPNNDV3zS1yJQVkkYGZYRmY5/eHP87rCCjLIHi8P9c11+U85znP+c4MDLfnPOcchRBCgIiIiEimbCq6ACIiIqLyxLBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsEOVyvr166FQKHDt2rWKLqVIgwcPRmBgoNXH3bdvHxQKBfbt22f1sel/EhMT8dJLL8HDwwMKhQJLliyxyrgffvghFAqFVcaiksv/7jh27FhFl1Js5VEzfw4Lx7DzhDp37hxeffVVVKlSBUqlEv7+/hg4cCDOnTtX0aXJxvPPP4+GDRsWuuzatWtQKBRYsGBBmbczZ84cbN++vczjPCnGjRuHXbt2YfLkydi0aRM6d+78wP5ZWVlYvHgxmjVrBo1GA5VKhdq1a2P06NH4+++/H1HVVFkMHjwYarW6osugErKr6ALo0YuJicGAAQPg7u6OoUOHIigoCNeuXcO6deuwbds2bNmyBb169aroMp84rVu3RmZmJhwcHEq03pw5c/DSSy+hZ8+e5VOYzPz666/o0aMHJkyY8NC+ycnJ6Ny5M44fP45u3brhlVdegVqtxqVLl7BlyxasWbMGOTk5j6BqIioLhp0nzOXLl/Haa6+hRo0a2L9/P7y8vKRlY8aMQatWrfDaa6/hzJkzqFGjxiOry2AwwNnZ+ZFtrzKysbGBSqWq6DJKRAiBrKwsODo6VnQpxXb79m1otdpi9R08eDBOnjyJbdu2oU+fPhbLZs2ahQ8++KAcKiQia+NhrCfM/PnzYTQasWbNGougAwCenp5YvXo1DAYD5s2bBwDYtm0bFAoFfvvttwJjrV69GgqFAn/99ZfUdvHiRbz00ktwd3eHSqXCM888g++++85ivfzj1L/99htGjhwJb29vVK1atciav/32W3Tt2hX+/v5QKpWoWbMmZs2ahby8PKnP9OnTYW9vj6SkpALrDx8+HFqtFllZWVLbjh070KpVKzg7O8PFxQVdu3Yt9BDe9u3b0bBhQ6hUKjRs2BDffPNNkXWWVWFzdmJjY9GnTx/4+vpCpVKhatWq6N+/P3Q6HQBAoVDAYDBgw4YNUCgUUCgUGDx4sLT+yZMn0aVLF7i6ukKtVqN9+/Y4fPhwgW2fOXMGbdq0gaOjI6pWrYrZs2cjOjq6wPypwMBAdOvWDbt27cIzzzwDR0dHrF69GgAQHR2Ndu3awdvbG0qlEvXr18eqVasKbCt/jH379kljhISESK87JiYGISEhUKlUaNKkCU6ePFms9+/KlSt4+eWX4e7uDicnJzz33HP48ccfpeX5P3dCCKxYsUJ6v4py5MgR/Pjjjxg6dGiBoAMASqXyoYchc3NzMWvWLNSsWRNKpRKBgYF4//33kZ2dbdHv2LFjCA8Ph6enJxwdHREUFITXX3/doo/ZbMaSJUvQoEEDqFQq+Pj4YMSIEbhz545Fv/z398CBA2jatClUKhVq1KiBjRs3FqgvLS0NY8eORUBAAJRKJWrVqoW5c+fCbDZb9NuyZQuaNGkCFxcXuLq6IiQkBJ988om03GQyYcaMGQgODoZKpYKHhwdatmyJ3bt3W4xTnO+H4o5VFKPRiBEjRsDDwwOurq4YNGiQxXsUEREBT09PmEymAut26tQJderUKdZ2HuT69esYOXIk6tSpA0dHR3h4eODll18uci7iw2rOV9zvLSqEoCeKv7+/CAwMfGCfwMBAUbVqVSGEEEajUajVajFy5MgC/dq2bSsaNGggPf/rr7+ERqMR9evXF3PnzhXLly8XrVu3FgqFQsTExEj9oqOjBQBRv3590aZNG7Fs2TLx8ccfWyy7evWq1L9nz56ib9++Yv78+WLVqlXi5ZdfFgDEhAkTpD6xsbECgFi2bJlFjdnZ2cLNzU28/vrrUtvGjRuFQqEQnTt3FsuWLRNz584VgYGBQqvVWmx3165dwsbGRjRs2FAsWrRIfPDBB0Kj0YgGDRqI6tWrP/A9FEKINm3aiLp164qkpKQCjxMnTggAYv78+VL/vXv3CgBi7969Uu1BQUHC399fzJ49W3z66adixowZ4tlnnxXXrl0TQgixadMmoVQqRatWrcSmTZvEpk2bxB9//CF9Hs7OzsLPz0/MmjVLfPzxxyIoKEgolUpx+PBhabv//vuvcHd3Fx4eHmLGjBliwYIFom7duiI0NLTAZ1G9enVRq1Yt4ebmJt577z0RFRUl1fvss8+KwYMHi8WLF4tly5aJTp06CQBi+fLlFu9L9erVRZ06dYSfn5/48MMPxeLFi0WVKlWEWq0Wn3/+uahWrZr4+OOPxccffyw0Go2oVauWyMvLe+B7nZCQIHx8fISLi4v44IMPxKJFi0RoaKiwsbGRfvYuX74sNm3aJACIjh07Su9XUd5//30BQOzfv//BH/T/mz59urj/KzUiIkIAEC+99JJYsWKFGDRokAAgevbsKfVJTEwUbm5uonbt2mL+/Pli7dq14oMPPhD16tWzGOuNN94QdnZ2YtiwYSIqKkq8++67wtnZWTz77LMiJyenwPvr4+Mj3n//fbF8+XLRuHFjoVAoxF9//SX1MxgM4qmnnhIeHh7i/fffF1FRUWLQoEFCoVCIMWPGSP1+/vlnAUC0b99erFixQqxYsUKMHj1avPzyyxbvlUKhEMOGDRNr164VCxcuFAMGDJB+r4Uo/vdDccYqTP53R0hIiGjVqpVYunSpGDVqlLCxsRGtW7cWZrNZCCHE7t27BQDx/fffW6wfHx8vbG1txcyZMx+4nYiICOHs7PzAPlu3bhWhoaFi2rRpYs2aNeL9998Xbm5uonr16sJgMJS4ZiGK/71V2M8hCcF35AmSlpYmAIgePXo8sN+LL74oAIj09HQhhBADBgwQ3t7eIjc3V+oTHx8vbGxsLL4Y2rdvL0JCQkRWVpbUZjabRfPmzUVwcLDUlv8L3rJlS4sx71127y+v0WgsUOOIESOEk5OTxbbCwsJEs2bNLPrFxMRYBIiMjAyh1WrFsGHDLPolJCQIjUZj0d6oUSPh5+cn0tLSpLb8L/7ihh0AD3w8KOycPHlSABBbt2594HacnZ1FREREgfaePXsKBwcHcfnyZant1q1bwsXFRbRu3Vpqe+utt4RCoRAnT56U2lJSUoS7u3uhYQeA2LlzZ4HtFfY5hYeHixo1ali05Y+RH8qEuBssAQhHR0dx/fp1qX316tUW70lRxo4dKwCI33//XWrLyMgQQUFBIjAw0CIsARCjRo164HhCCNGrVy8BQNy5c+ehfYUo+Efm1KlTAoB44403LPpNmDBBABC//vqrEEKIb775RgAQR48eLXLs33//XQAQX3zxhUX7zp07C7Tnv7/3hrTbt28LpVIp3nnnHalt1qxZwtnZWfz9998WY7733nvC1tZW3LhxQwghxJgxY4Srq2uB39V7hYaGiq5duxa5XIjifz8UZ6zC5H93NGnSxCL8zZs3TwAQ3377rRBCiLy8PFG1alXRr18/i/UXLVokFAqFuHLlygO3U5ywU9jvwqFDhwQAsXHjxhLXXJLvLYadwvEw1hMkIyMDAODi4vLAfvnL09PTAQD9+vXD7du3LQ6vbNu2DWazGf369QMApKam4tdff0Xfvn2RkZGB5ORkJCcnIyUlBeHh4YiNjcXNmzcttjNs2DDY2to+tO5754Pkj92qVSsYjUZcvHhRWjZo0CAcOXIEly9fltq++OILBAQEoE2bNgCA3bt3Iy0tDQMGDJBqTE5Ohq2tLZo1a4a9e/cCAOLj43Hq1ClERERAo9FI43Xs2BH169d/aM35AgMDsXv37gKPzz///KHr5m93165dMBqNxd4mAOTl5eHnn39Gz549LeZe+fn54ZVXXsGBAwekz3fnzp0ICwtDo0aNpH7u7u4YOHBgoWMHBQUhPDy8QPu9n5NOp0NycjLatGmDK1euSIfd8tWvXx9hYWHS82bNmgEA2rVrh2rVqhVov3LlygNf708//YSmTZuiZcuWUptarcbw4cNx7do1nD9//oHrFyb//XnY78uDagKA8ePHW7S/8847ACAdYsufP/TDDz8UemgFALZu3QqNRoOOHTta/Nw2adIEarVa+rnNV79+fbRq1Up67uXlhTp16li8j1u3bkWrVq3g5uZmMWaHDh2Ql5eH/fv3S/UZDIYHHkbSarU4d+4cYmNjC11eku+Hh431MMOHD4e9vb30/D//+Q/s7Oykz8PGxgYDBw7Ed999J30nAne/K5o3b46goKBSbfde9/4umEwmpKSkoFatWtBqtThx4kSJay7u9xYVjWHnCZL/pX3vL3hh7g9FnTt3hkajwVdffSX1+eqrr9CoUSPUrl0bAPDPP/9ACIGpU6fCy8vL4jF9+nQAdyeG3qu4Xyrnzp1Dr169oNFo4OrqCi8vL7z66qsAYPFHtF+/flAqlfjiiy+kZT/88AMGDhwozc3I/wJt165dgTp//vlnqcbr168DAIKDgwvUU5Jj+s7OzujQoUOBR4sWLR66blBQEMaPH49PP/0Unp6eCA8Px4oVKwoEh8IkJSXBaDQWWmu9evVgNpsRFxcH4O5rrVWrVoF+hbXl11WYgwcPokOHDnB2doZWq4WXlxfef/99AChQ872BBvhfsAsICCi0vbD5C/e6fv16ka81f3lJubq6Anj478uDarKxsSnwPvr6+kKr1Uo1tWnTBn369MGMGTPg6emJHj16IDo62mJeT2xsLHQ6Hby9vQv83Or1+gK/W/e/vwDg5uZm8T7GxsZi586dBcbr0KEDgP/9vo4cORK1a9dGly5dULVqVbz++uvYuXOnxdgzZ85EWloaateujZCQEEycOBFnzpyRlpfk++FhYz3M/b+zarUafn5+FvNlBg0ahMzMTGkO3qVLl3D8+HG89tprxd7Og2RmZmLatGnSXChPT094eXkhLS2t0N/fh9Vc3O8tKhrPxnqCaDQa+Pn5PfSL48yZM6hSpYr0Za9UKtGzZ0988803WLlyJRITE3Hw4EHMmTNHWid/QuOECRMK/V8/UPCPZ3HO4ElLS0ObNm3g6uqKmTNnombNmlCpVDhx4gTeffddi4mUbm5u6NatG7744gtMmzYN27ZtQ3Z2thSM7q1z06ZN8PX1LbA9O7vK9SuxcOFCDB48GN9++y1+/vlnvP3224iMjMThw4cfOKm7PBX2uV2+fBnt27dH3bp1sWjRIgQEBMDBwQE//fQTFi9eXGDCa1F79IpqF0KUvfASqlu3LgDg7NmzFntJSuphF3hTKBTYtm0bDh8+jO+//x67du3C66+/joULF+Lw4cNQq9Uwm83w9vaWgvz97j/ZoDjvo9lsRseOHTFp0qRC++b/R8bb2xunTp3Crl27sGPHDuzYsQPR0dEYNGgQNmzYAODuZRMuX74s/Zx++umnWLx4MaKiovDGG2+U6PvhYWNZQ/369dGkSRN8/vnnGDRoED7//HM4ODigb9++Vhn/rbfeQnR0NMaOHYuwsDBoNBooFAr079+/wO9CcTxu31uVEd+hJ0y3bt2wdu1aHDhwwGKXf77ff/8d165dw4gRIyza+/Xrhw0bNmDPnj24cOEChBDSISwA0qESe3t76X+G1rBv3z6kpKQgJiYGrVu3ltqvXr1aaP9BgwahR48eOHr0KL744gs8/fTTaNCggbS8Zs2aAO5+gT+ozurVqwNAobvSL126VKrXUlohISEICQnBlClT8Mcff6BFixaIiorC7NmzART+x9TLywtOTk6F1nrx4kXY2NhIe1GqV6+Of/75p0C/wtqK8v333yM7OxvfffedxV6FR7V7vXr16kW+1vzlJdW9e3dERkbi888/L1XYqV69OsxmM2JjY6U9TMDdKzinpaUVqOm5557Dc889h48++gibN2/GwIEDsWXLFrzxxhuoWbMmfvnlF7Ro0cJqp/nXrFkTer2+WL+vDg4O6N69O7p37w6z2YyRI0di9erVmDp1qhRS3N3dMWTIEAwZMgR6vR6tW7fGhx9+iDfeeKPE3w8PGuthYmNj0bZtW+m5Xq9HfHw8XnjhBYt+gwYNwvjx4xEfH4/Nmzeja9eucHNze+j4xbFt2zZERERg4cKFUltWVhbS0tJKVXNxv7eoaDyM9YSZOHEiHB0dMWLECKSkpFgsS01NxZtvvgknJydMnDjRYlmHDh3g7u6Or776Cl999RWaNm1qcTjD29sbzz//PFavXo34+PgC2y3slPDiyP8f6r3/I83JycHKlSsL7d+lSxd4enpi7ty5+O233yz26gBAeHg4XF1dMWfOnELnR+TX6efnh0aNGmHDhg0Wu513795dqvkfpZGeno7c3FyLtpCQENjY2Fgc4nB2di7wJWpra4tOnTrh22+/tdh9n5iYiM2bN6Nly5bSnrvw8HAcOnQIp06dkvqlpqYWuRehMIV9TjqdDtHR0cUeoyxeeOEF/Pnnnzh06JDUZjAYsGbNGgQGBpZonlW+sLAwdO7cGZ9++mmhV6jOycl54IUJ8/9Q3X87ikWLFgEAunbtCuDuIbr791zlz5/K/5z79u2LvLw8zJo1q8B2cnNzi/wj+iB9+/bFoUOHsGvXrgLL0tLSpJ+9+78nbGxs8NRTT1nUd38ftVqNWrVqSctL8v3wsLEeZs2aNRa/26tWrUJubi66dOli0W/AgAFQKBQYM2YMrly5UuC7oixsbW0LfKbLli2zuFxGSWou7vcWFY17dp4wwcHB2LBhAwYOHIiQkJACV1BOTk7Gl19+Kf1PIp+9vT169+6NLVu2wGAwFHp9kRUrVqBly5YICQnBsGHDUKNGDSQmJuLQoUP4999/cfr06RLX27x5c7i5uSEiIgJvv/02FAoFNm3aVORhDXt7e/Tv3x/Lly+Hra0tBgwYYLHc1dUVq1atwmuvvYbGjRujf//+8PLywo0bN/Djjz+iRYsWWL58OQAgMjISXbt2RcuWLfH6668jNTUVy5YtQ4MGDaDX60v8Wkrq119/xejRo/Hyyy+jdu3ayM3NxaZNm2Bra2tx3ZcmTZrgl19+waJFi+Dv74+goCA0a9YMs2fPxu7du9GyZUuMHDkSdnZ2WL16NbKzs6XrKAHApEmT8Pnnn6Njx45466234OzsjE8//RTVqlVDampqse6z06lTJ+l//yNGjIBer8fatWvh7e1d6B83a3vvvffw5ZdfokuXLnj77bfh7u6ODRs24OrVq/jvf/8LG5vS/b9u48aN6NSpE3r37o3u3bujffv2cHZ2RmxsLLZs2YL4+Pgir7UTGhqKiIgIrFmzRjoc++eff2LDhg3o2bOn9D/5DRs2YOXKlejVqxdq1qyJjIwMrF27Fq6urlJgatOmDUaMGIHIyEicOnUKnTp1gr29PWJjY7F161Z88skneOmll0r02iZOnIjvvvsO3bp1w+DBg9GkSRMYDAacPXsW27Ztw7Vr1+Dp6Yk33ngDqampaNeuHapWrYrr169j2bJlaNSokbTHqn79+nj++efRpEkTuLu749ixY9i2bRtGjx4tba+43w/FGetBcnJy0L59e/Tt2xeXLl3CypUr0bJlS7z44osW/by8vNC5c2ds3boVWq1WCp/FYTKZpD2r93J3d8fIkSPRrVs3bNq0CRqNBvXr18ehQ4fwyy+/wMPDo1Q1l+R7i4pQUaeBUcU6c+aMGDBggPDz8xP29vbC19dXDBgwQJw9e7bIdfKvT6FQKERcXFyhfS5fviwGDRokfH19hb29vahSpYro1q2b2LZtm9Qn/3TLwk61LezU84MHD4rnnntOODo6Cn9/fzFp0iTpVOXCTkn+888/BQDRqVOnIl/L3r17RXh4uNBoNEKlUomaNWuKwYMHi2PHjln0++9//yvq1asnlEqlqF+/voiJiRERERHFPvX83usQ3evq1asPPfX8ypUr4vXXXxc1a9YUKpVKuLu7i7Zt24pffvnFYqyLFy+K1q1bC0dHRwHA4jT0EydOiPDwcKFWq4WTk5No27atxSnf+U6ePClatWollEqlqFq1qoiMjBRLly4VAERCQoLUr3r16kWeFvzdd9+Jp556SqhUKhEYGCjmzp0rPvvss0JPXy9sDBRySnhh71NRLl++LF566SWh1WqFSqUSTZs2FT/88EOxtvMgRqNRLFiwQDz77LNCrVYLBwcHERwcLN566y3xzz//SP0KO+XXZDKJGTNmiKCgIGFvby8CAgLE5MmTLU6/PnHihBgwYICoVq2aUCqVwtvbW3Tr1q3Az6IQQqxZs0Y0adJEODo6ChcXFxESEiImTZokbt26JfUp6v1t06aNaNOmjUVbRkaGmDx5sqhVq5ZwcHAQnp6eonnz5mLBggXSqdDbtm0TnTp1Et7e3sLBwUFUq1ZNjBgxQsTHx0vjzJ49WzRt2lRotVrh6Ogo6tatKz766COL06mFKN73Q3HHul/+d8dvv/0mhg8fLtzc3IRarRYDBw4UKSkpha7z9ddfCwBi+PDhDxz7XvnXTirsUbNmTSGEEHfu3BFDhgwRnp6eQq1Wi/DwcHHx4kVRvXp1i9/PktZcnO8tnnpeOIUQFTDzj6gcnT59Go0aNcLGjRutdnbFk2js2LFYvXo19Hp9sS4RQPS4+fbbb9GzZ0/s37+/TJPQqfLjnB2SnbVr10KtVqN3794VXcpjIzMz0+J5SkoKNm3ahJYtWzLokGytXbsWNWrUKPRkDZIXztkh2fj+++9x/vx5rFmzBqNHj37ibyxaEmFhYXj++edRr149JCYmYt26dUhPT8fUqVMrujQiq9uyZQvOnDmDH3/8EZ988kmx5qXR442HsUg2AgMDkZiYiPDwcGzatKnUV759Er3//vvYtm0b/v33XygUCjRu3BjTp0/naa4kSwqFAmq1Gv369UNUVBSvU/MEYNghIiIiWeOcHSIiIpI1hh0iIiKSNR6oxN37jty6dQsuLi6cqEZERPSYEEIgIyMD/v7+D7x4KMMOgFu3bhW42zIRERE9HuLi4h54c2SGHUA6aycuLk66XxARERFVbunp6QgICHjo2bcMO/jfXaNdXV0ZdoiIiB4zD5uCwgnKREREJGsMO0RERCRrDDtEREQka5yzQ0REZZaXlweTyVTRZZDM2NvbW+VmxAw7RERUakIIJCQkIC0traJLIZnSarXw9fUt03XwGHaIiKjU8oOOt7c3nJyceGFWshohBIxGI27fvg0A8PPzK/VYDDtERFQqeXl5UtDx8PCo6HJIhhwdHQEAt2/fhre3d6kPaXGCMhERlUr+HB0nJ6cKroTkLP/nqyxzwhh2iIioTHjoisqTNX6+GHaIiIhI1jhnh4iIrE6n08FoND6y7Tk5OUGj0Tyy7T1uBg8ejLS0NGzfvr3UY+zbtw9t27bFnTt3oNVqrVbbo8CwQ0REVqXT6fDRvMVIyXh0YcfDxQkfTBpX7MCTlJSEadOm4ccff0RiYiLc3NwQGhqKadOmoUWLFuVcbckFBgZi7NixGDt2bEWX8lhi2CEiIqsyGo1IyTDCvUFLqDXu5b49vS4VKecOwGg0Fjvs9OnTBzk5OdiwYQNq1KiBxMRE7NmzBykpKeVcLVUEztkhIqJyoda4w9XDu9wfJQ1UaWlp+P333zF37ly0bdsW1atXR9OmTTF58mS8+OKLUr9FixYhJCQEzs7OCAgIwMiRI6HX66Xl69evh1arxQ8//IA6derAyckJL730EoxGIzZs2IDAwEC4ubnh7bffRl5enrRednY2JkyYgCpVqsDZ2RnNmjXDvn37Sv0+5+XlYejQoQgKCoKjoyPq1KmDTz75pNC+M2bMgJeXF1xdXfHmm28iJydHWmY2mxEZGSmNExoaim3btpW6rsqEe3aIHjPlNReCcx7oSaFWq6FWq7F9+3Y899xzUCqVhfazsbHB0qVLERQUhCtXrmDkyJGYNGkSVq5cKfUxGo1YunQptmzZgoyMDPTu3Ru9evWCVqvFTz/9hCtXrqBPnz5o0aIF+vXrBwAYPXo0zp8/jy1btsDf3x/ffPMNOnfujLNnzyI4OLjEr8dsNqNq1arYunUrPDw88Mcff2D48OHw8/ND3759pX579uyBSqXCvn37cO3aNQwZMgQeHh746KOPAACRkZH4/PPPERUVheDgYOzfvx+vvvoqvLy80KZNmxLXVZkw7BA9RnQ6HZbPnw1TRrLVx7Z38cToiVMYeEj27OzssH79egwbNgxRUVFo3Lgx2rRpg/79++Opp56S+t07PyYwMBCzZ8/Gm2++aRF2TCYTVq1ahZo1awIAXnrpJWzatAmJiYlQq9WoX78+2rZti71796Jfv364ceMGoqOjcePGDfj7+wMAJkyYgJ07dyI6Ohpz5swp8euxt7fHjBkzpOdBQUE4dOgQvv76a4uw4+DggM8++wxOTk5o0KABZs6ciYkTJ2LWrFkwmUyYM2cOfvnlF4SFhQEAatSogQMHDmD16tUMO0T06BiNRpgyktE7xAVeWmerjZuUZkDM2eQSzXkgepz16dMHXbt2xe+//47Dhw9jx44dmDdvHj799FMMHjwYAPDLL78gMjISFy9eRHp6OnJzc5GVlQWj0Shd6M7JyUkKOgDg4+ODwMBAqNVqi7b8Wx6cPXsWeXl5qF27tkU92dnZZboK9YoVK/DZZ5/hxo0byMzMRE5ODho1amTRJzQ01OICkGFhYdDr9YiLi4Ner4fRaETHjh0t1snJycHTTz9d6roqC4YdoseQl9YZfh6uVh41w8rjEVVuKpUKHTt2RMeOHTF16lS88cYbmD59OgYPHoxr166hW7du+M9//oOPPvoI7u7uOHDgAIYOHYqcnBwpNNjb21uMqVAoCm0zm80AAL1eD1tbWxw/frzArQ/uDUglsWXLFkyYMAELFy5EWFgYXFxcMH/+fBw5cqTYY+TPRfrxxx9RpUoVi2VFHeZ7nDDsEBERAahfv750HZrjx4/DbDZj4cKFsLG5ey7P119/XeZtPP3008jLy8Pt27fRqlWrMo8HAAcPHkTz5s0xcuRIqe3y5csF+p0+fRqZmZnS/aYOHz4MtVqNgIAAuLu7Q6lU4saNG4/9IavCMOwQEdETJSUlBS+//DJef/11PPXUU3BxccGxY8cwb9489OjRAwBQq1YtmEwmLFu2DN27d8fBgwcRFRVV5m3Xrl0bAwcOxKBBg7Bw4UI8/fTTSEpKwp49e/DUU0+ha9euRa578+ZNnDp1yqKtevXqCA4OxsaNG7Fr1y4EBQVh06ZNOHr0KIKCgiz65uTkYOjQoZgyZQquXbuG6dOnY/To0bCxsYGLiwsmTJiAcePGwWw2o2XLltDpdDh48CBcXV0RERFR5tdekRh2iIioXOh1qZVyO2q1Gs2aNcPixYtx+fJlmEwmBAQEYNiwYXj//fcB3J3fsmjRIsydOxeTJ09G69atERkZiUGDBpW53ujoaMyePRvvvPMObt68CU9PTzz33HPo1q3bA9dbsGABFixYYNG2adMmjBgxAidPnkS/fv2gUCgwYMAAjBw5Ejt27LDo2759ewQHB6N169bIzs7GgAED8OGHH0rLZ82aBS8vL0RGRuLKlSvQarVo3Lix9J48zhRCCFHRRVS09PR0aDQa6HQ6uLpaex4EkfXEx8dj9cfvY0QrX6vO2YlPScfq3xMw4r058PPzs9q4JG9ZWVm4evUqgoKCoFKppPbH4QrK9Pgo6ucMKP7fb+7ZISIiq9JoNPhg0jjeG4sqDYYdIiKyOo1Gw/BBlQZvF0FERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREslahYWf//v3o3r07/P39oVAopMt0A3fvJPvuu+8iJCQEzs7O8Pf3x6BBg3Dr1i2LMVJTUzFw4EC4urpCq9Vi6NCh0j0+iIiIiCo07BgMBoSGhmLFihUFlhmNRpw4cQJTp07FiRMnEBMTg0uXLuHFF1+06Ddw4ECcO3cOu3fvxg8//ID9+/dj+PDhj+olEBERUSVXodfZ6dKlC7p06VLoMo1Gg927d1u0LV++HE2bNsWNGzdQrVo1XLhwATt37sTRo0fxzDPPAACWLVuGF154AQsWLIC/v3+5vwai8qLT6QpclC0xMRHGzEzoDQZkKBVW25beYECOyWS18YgK+/ktT+V5UUGFQoFvvvkGPXv2BABcvHgRgwcPxqlTp1C3bl2cOnWq0LYnweDBg5GWlmZxZKak9u3bh7Zt2+LOnTvQarVWq+1ej9VFBXU6HRQKhfRmHDp0CFqtVgo6ANChQwfY2NjgyJEj6NWrVwVVSlQ2RV1u32jQI/n8eRywj4O7Wmm17aXqs3HmLyPS09N5uwgqM51Oh+XzZ8OUkfzItmnv4onRE6cUO/AMHjwYGzZsAADY2dnB3d0dTz31FAYMGIDBgwdLdzoH7t6mxc3NTXo+ffp0ODs749KlS1Cr1UW2VaTAwECMHTsWY8eOtUq/x91jE3aysrLw7rvvYsCAAdL9LxISEuDt7W3RL/+HNiEhocixsrOzkZ2dLT1PT08vn6KJSsloNCIlwwj3Bi2h1rhL7Rl3kpGZdB2ufh7QaJytt72kFGSbLiIzM9NqY9KTy2g0wpSRjN4hLvDSWu/ntChJaQbEnE2G0Wgs0d6dzp07Izo6Gnl5eUhMTMTOnTsxZswYbNu2Dd999x3s7O7+ifT19bVY7/Lly+jatSuqV6/+wLaSysnJgYODQ6nXp6I9FmdjmUwm9O3bF0IIrFq1qszjRUZGSpcy12g0CAgIsEKVRNan1rjD1cNberi4ecJepYKDoxOUTs5Wezg4qB5eDFEJeWmd4efhWu6P0gYqpVIJX19fVKlSRbq797fffosdO3Zg/fr1Ur97T6BRKBQ4fvw4Zs6cCYVCgQ8//LDQNgCIi4tD3759odVq4e7ujh49euDatWvSuIMHD0bPnj3x0Ucfwd/fH3Xq1CnRegsWLICfnx88PDwwatQomP7/UPTzzz+P69evY9y4cVAoFFAoSnfIOy8vD0OHDkVQUBAcHR1Rp04dfPLJJ4X2nTFjBry8vODq6oo333wTOTk50jKz2YzIyEhpnNDQUGzbtq1UNZVWpQ87+UHn+vXr2L17t8VdTX19fXH79m2L/rm5uUhNTS2QxO81efJk6HQ66REXF1du9RMR0eOjXbt2CA0NRUxMTKHL4+Pj0aBBA7zzzjuIj4/HhAkTCm0zmUwIDw+Hi4sLfv/9dxw8eBBqtRqdO3e2CAJ79uzBpUuXpJNsirve3r17cfnyZezduxcbNmzA+vXrpYAWExODqlWrYubMmYiPj0d8fHyp3guz2YyqVati69atOH/+PKZNm4b3338fX3/9tUW/PXv24MKFC9i3bx++/PJLxMTEYMaMGdLyyMhIbNy4EVFRUTh37hzGjRuHV199Fb/99lup6iqNSn0YKz/oxMbGYu/evfDw8LBYHhYWhrS0NBw/fhxNmjQBAPz6668wm81o1qxZkeMqlUooldab70BERPJRt25dnDlzptBlvr6+sLOzg1qtlv5TrVarC7R9/vnnMJvN+PTTT6U9K9HR0dBqtdi3bx86deoEAHB2dsann34qHb4q7npubm5Yvnw5bG1tUbduXXTt2hV79uzBsGHD4O7uDltbW7i4uDzwP/4PY29vbxFagoKCcOjQIXz99dfo27ev1O7g4IDPPvsMTk5OaNCgAWbOnImJEydi1qxZMJlMmDNnDn755ReEhYUBAGrUqIEDBw5g9erVaNOmTanrK4kKDTt6vR7//POP9Pzq1as4deoU3N3d4efnh5deegknTpzADz/8gLy8PGkejru7OxwcHFCvXj107twZw4YNQ1RUFEwmE0aPHo3+/fvzTCwiIioVIUSpD/3kO336NP755x+4uLhYtGdlZeHy5cvS85CQEIt5OsVdr0GDBrC1tZWe+/n54ezZs2WquTArVqzAZ599hhs3biAzMxM5OTlo1KiRRZ/Q0FA4OTlJz8PCwqDX6xEXFwe9Xg+j0YiOHTtarJOTk4Onn37a6vUWpULDzrFjx9C2bVvp+fjx4wEAERER+PDDD/Hdd98BQIE3du/evXj++ecBAF988QVGjx6N9u3bw8bGBn369MHSpUsfSf1ERCQ/Fy5cQFBQUJnG0Ov1aNKkCb744osCy7y8vKR/Oztbzjcq7nr29vYWyxQKBcxmc5lqvt+WLVswYcIELFy4EGFhYXBxccH8+fNx5MiRYo+Rf5HfH3/8EVWqVLFY9iiPsFRo2Hn++echhChy+YOW5XN3d8fmzZutWRYRET2hfv31V5w9exbjxo0r0ziNGzfGV199BW9vb4u5puW13v0cHByQl5dX6vUB4ODBg2jevDlGjhwptd27dynf6dOnkZmZCUdHRwDA4cOHoVarERAQAHd3dyiVSty4ceORHbIqTKWfoExERFQesrOzkZCQgJs3b+LEiROYM2cOevTogW7dumHQoEFlGnvgwIHw9PREjx498Pvvv+Pq1avYt28f3n77bfz7779WX+9+gYGB2L9/P27evInk5Adf7+jmzZs4deqUxePOnTsIDg7GsWPHsGvXLvz999+YOnUqjh49WmD9nJwcDB06FOfPn8dPP/2E6dOnY/To0bCxsYGLiwsmTJiAcePGYcOGDbh8+TJOnDiBZcuWSdc5ehQq9QRlIiJ6fCWlGSr1dnbu3Ak/Pz/Y2dnBzc0NoaGhWLp0KSIiIiwuKlgaTk5O2L9/P95991307t0bGRkZqFKlCtq3b//APTalXe9+M2fOxIgRI1CzZk1kZ2c/8EjJggULsGDBAou2TZs2YcSIETh58iT69esHhUKBAQMGYOTIkdixY4dF3/bt2yM4OBitW7dGdnY2BgwYIJ1+DwCzZs2Cl5cXIiMjceXKFWi1WulU/0dFIYpzrEjm0tPTodFooNPpyrTbkMha4uPjMSVyMao1fxGuHv+7cGZ6ym38FbMYg5p5wVNrvau03oq/jagfT2H+qo0F5sgRFSUrKwtXr15FUFAQVKr/XavpcbiCMj0+ivo5A4r/95t7doiIyKo0Gg1GT5wim3tj0eOPYYeIiKwu/wr1RJUBJygTERGRrDHsEBERkawx7BARUZnwPBcqT9b4+WLYISKiUsm/iu+jnIhMT578n6/7rxpdEpygTEREpWJrawutVovbt28DuHtGVFnvKUWUTwgBo9GI27dvQ6vVWtwLrKQYdoiIqNTy76qdH3iIrE2r1Zbp7u0Aww4REZWBQqGAn58fvL29YTKZKrockhl7e/sy7dHJx7BDRERlZmtra5U/SkTlgROUiYiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNYYdoiIiEjWGHaIiIhI1hh2iIiISNbsKroAIjnT6XQwGo0lXi8xMRFGgx4Zd5It2jPuJCPPZLJWeURETwSGHaJyotPpsHz+bJgykh/e+T7GzEwknz+PzKTrsFeppPbMzEyk37yM3Gae1iyViEjWGHaIyonRaIQpIxm9Q1zgpXUu0bp6gwEH7OPg6ucBB0cnqf3arRRsi8uGOTfX2uUSEckWww5ROfPSOsPPw7VE62QoFXBXK6HROEPp9L+glJpusHZ5RESyV6ETlPfv34/u3bvD398fCoUC27dvt1guhMC0adPg5+cHR0dHdOjQAbGxsRZ9UlNTMXDgQLi6ukKr1WLo0KHQ6/WP8FUQERFRZVahYcdgMCA0NBQrVqwodPm8efOwdOlSREVF4ciRI3B2dkZ4eDiysrKkPgMHDsS5c+ewe/du/PDDD9i/fz+GDx/+qF4CERERVXIVehirS5cu6NKlS6HLhBBYsmQJpkyZgh49egAANm7cCB8fH2zfvh39+/fHhQsXsHPnThw9ehTPPPMMAGDZsmV44YUXsGDBAvj7+z+y10JERESVU6W9zs7Vq1eRkJCADh06SG0ajQbNmjXDoUOHAACHDh2CVquVgg4AdOjQATY2Njhy5EiRY2dnZyM9Pd3iQURERPJUacNOQkICAMDHx8ei3cfHR1qWkJAAb29vi+V2dnZwd3eX+hQmMjISGo1GegQEBFi5eiIiIqosKm3YKU+TJ0+GTqeTHnFxcRVdEhEREZWTSht2fH19Ady9kuy9EhMTpWW+vr64ffu2xfLc3FykpqZKfQqjVCrh6upq8SAiIiJ5qrRhJygoCL6+vtizZ4/Ulp6ejiNHjiAsLAwAEBYWhrS0NBw/flzq8+uvv8JsNqNZs2aPvGYiIiKqfCr0bCy9Xo9//vlHen716lWcOnUK7u7uqFatGsaOHYvZs2cjODgYQUFBmDp1Kvz9/dGzZ08AQL169dC5c2cMGzYMUVFRMJlMGD16NPr3788zsYiIiAhABYedY8eOoW3bttLz8ePHAwAiIiKwfv16TJo0CQaDAcOHD0daWhpatmyJnTt3QnXPvYK++OILjB49Gu3bt4eNjQ369OmDpUuXPvLXQkRERJVThYad559/HkKIIpcrFArMnDkTM2fOLLKPu7s7Nm/eXB7lERERkQxU2jk7RERERNbAsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLJmV9EFEFHlkJubh6SkJMTHx1ttTCcnJ2g0GquNR0RUGgw7RARDVg7upCTh+43LcdDd3Wrj2rt4YvTEKQw8RFShGHaICDmmXDja5qFnQzXq1fS1yphJaQbEnE2G0Whk2CGiCsWwQ0QSDxdH+Hm4WnHEDCuORURUOpygTERERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssYbgRIRAEAIM4yZRmTorXPzTr3BAGNmJhITEwtd7uTkxLuhE9EjwbBDRMjNzUVOdg5OnbuI27cTrDJmqj4bJ89mIG7ZGjg5qwss93BxwgeTxjHwEFG5Y9ghIpjzciEAKN18oala3Spj5uoMcPZKQZVnOsHFzdNimV6XipRzB2A0Ghl2iKjcMewQkcTeXgmlk7NVxnLIEbBXGeDi5glXD+8Cy1OtshUioofjBGUiIiKSNYYdIiIikjWGHSIiIpI1hh0iIiKSNU5QJiomnU4Ho9FY7P6JiYkwZmZCbzAgQ6ko0bYMegPMeXklLZGIiArBsENUDDqdDh/NW4yUjOKHHaNBj+Tz53HAPg7uamWJtmfKyUFySipcq+aWtFQiIroPww5RMRiNRqRkGOHeoCXUGvdirZNxJxmZSdfh6ucBjaZkp3PrdanITUqG2WwuTblERHQPhh2iElBr3Au9ZkxR7FUqODg6lfjaNTlZxd+DRERED1apJyjn5eVh6tSpCAoKgqOjI2rWrIlZs2ZBCCH1EUJg2rRp8PPzg6OjIzp06IDY2NgKrJqIiIgqk0oddubOnYtVq1Zh+fLluHDhAubOnYt58+Zh2bJlUp958+Zh6dKliIqKwpEjR+Ds7Izw8HBkZWVVYOVERERUWVTqw1h//PEHevToga5duwIAAgMD8eWXX+LPP/8EcHevzpIlSzBlyhT06NEDALBx40b4+Phg+/bt6N+/f4XVTkRERJVDpd6z07x5c+zZswd///03AOD06dM4cOAAunTpAgC4evUqEhIS0KFDB2kdjUaDZs2a4dChQxVSMxEREVUulXrPznvvvYf09HTUrVsXtra2yMvLw0cffYSBAwcCABISEgAAPj4+Fuv5+PhIywqTnZ2N7Oxs6Xl6eno5VE9ERESVQaXes/P111/jiy++wObNm3HixAls2LABCxYswIYNG8o0bmRkJDQajfQICAiwUsVERERU2VTqsDNx4kS899576N+/P0JCQvDaa69h3LhxiIyMBAD4+voCuHul2nslJiZKywozefJk6HQ66REXF1d+L4KIiIgqVKUOO0ajETY2liXa2tpKF1oLCgqCr68v9uzZIy1PT0/HkSNHEBYWVuS4SqUSrq6uFg8iIiKSp0o9Z6d79+746KOPUK1aNTRo0AAnT57EokWL8PrrrwMAFAoFxo4di9mzZyM4OBhBQUGYOnUq/P390bNnz4otnoiIiCqFUoWdK1euoEaNGtaupYBly5Zh6tSpGDlyJG7fvg1/f3+MGDEC06ZNk/pMmjQJBoMBw4cPR1paGlq2bImdO3dCpVKVe31ERERU+ZUq7NSqVQtt2rTB0KFD8dJLL5VbsHBxccGSJUuwZMmSIvsoFArMnDkTM2fOLJcaiIiI6PFWqjk7J06cwFNPPYXx48fD19cXI0aMkC70R0RERFSZlCrsNGrUCJ988glu3bqFzz77DPHx8WjZsiUaNmyIRYsWISkpydp1EhEREZVKmc7GsrOzQ+/evbF161bMnTsX//zzDyZMmICAgAAMGjQI8fHx1qqTiIiIqFTKFHaOHTuGkSNHws/PD4sWLcKECRNw+fJl7N69G7du3ZLuV0VERERUUUo1QXnRokWIjo7GpUuX8MILL2Djxo144YUXpGviBAUFYf369QgMDLRmrUREREQlVqqws2rVKrz++usYPHgw/Pz8Cu3j7e2NdevWlak4IiIiorIqVdiJjY19aB8HBwdERESUZngiIiIiqynVnJ3o6Ghs3bq1QPvWrVvLfJNOIiIiImsqVdiJjIyEp6dngXZvb2/MmTOnzEURERERWUupws6NGzcQFBRUoL169eq4ceNGmYsiIiIispZShR1vb2+cOXOmQPvp06fh4eFR5qKIiIiIrKVUYWfAgAF4++23sXfvXuTl5SEvLw+//vorxowZg/79+1u7RiIiIqJSK9XZWLNmzcK1a9fQvn172NndHcJsNmPQoEGcs0NERESVSqnCjoODA7766ivMmjULp0+fhqOjI0JCQlC9enVr10dERERUJqUKO/lq166N2rVrW6sWIiIiIqsrVdjJy8vD+vXrsWfPHty+fRtms9li+a+//mqV4oiIiIjKqlRhZ8yYMVi/fj26du2Khg0bQqFQWLsuIiIiIqsoVdjZsmULvv76a7zwwgvWroeIiIjIqkp16rmDgwNq1apl7VqIiIiIrK5UYeedd97BJ598AiGEteshIiIisqpSHcY6cOAA9u7dix07dqBBgwawt7e3WB4TE2OV4oiIiIjKqlRhR6vVolevXtauhYiIiMjqShV2oqOjrV0HERERUbko1ZwdAMjNzcUvv/yC1atXIyMjAwBw69Yt6PV6qxVHREREVFal2rNz/fp1dO7cGTdu3EB2djY6duwIFxcXzJ07F9nZ2YiKirJ2nURERESlUqo9O2PGjMEzzzyDO3fuwNHRUWrv1asX9uzZY7XiiIiIiMqqVHt2fv/9d/zxxx9wcHCwaA8MDMTNmzetUhgRERGRNZRqz47ZbEZeXl6B9n///RcuLi5lLoqIiIjIWkoVdjp16oQlS5ZIzxUKBfR6PaZPn85bSBAREVGlUqrDWAsXLkR4eDjq16+PrKwsvPLKK4iNjYWnpye+/PJLa9dIREREVGqlCjtVq1bF6dOnsWXLFpw5cwZ6vR5Dhw7FwIEDLSYsExEREVW0UoUdALCzs8Orr75qzVqIiIiIrK5UYWfjxo0PXD5o0KBSFUNERERkbaUKO2PGjLF4bjKZYDQa4eDgACcnJ4YdeizpdDoYjcZClyUmJsJo0CPjTnKxx8u4k4w8k8la5RERUSmVKuzcuXOnQFtsbCz+85//YOLEiWUuiuhR0+l0WD5/NkwZhYcZY2Ymks+fR2bSddirVMUaMzMzE+k3LyO3mac1SyUiohIq9Zyd+wUHB+Pjjz/Gq6++iosXL1prWKJHwmg0wpSRjN4hLvDSOhdYrjcYcMA+Dq5+HnBwdCrWmNdupWBbXDbMubnWLpeIiErAamEHuDtp+datW9YckuiR8tI6w8/DtUB7hlIBd7USGo0zlE4Fw1BhUtMN1i6PiIhKoVRh57vvvrN4LoRAfHw8li9fjhYtWlilMCIiIiJrKFXY6dmzp8VzhUIBLy8vtGvXDgsXLrRGXURERERWUaqwYzabrV0HERERUbko1b2xiIiIiB4XpdqzM378+GL3XbRoUWk2QURERGQVpQo7J0+exMmTJ2EymVCnTh0AwN9//w1bW1s0btxY6qdQKKxTJREREVEplSrsdO/eHS4uLtiwYQPc3NwA3L3Q4JAhQ9CqVSu88847Vi2SiIiIqLRKNWdn4cKFiIyMlIIOALi5uWH27Nk8G4uIiIgqlVKFnfT0dCQlJRVoT0pKQkZGRpmLutfNmzfx6quvwsPDA46OjggJCcGxY8ek5UIITJs2DX5+fnB0dESHDh0QGxtr1RqIiIjo8VWqsNOrVy8MGTIEMTEx+Pfff/Hvv//iv//9L4YOHYrevXtbrbg7d+6gRYsWsLe3x44dO3D+/HksXLjQYo/SvHnzsHTpUkRFReHIkSNwdnZGeHg4srKyrFYHERERPb5KNWcnKioKEyZMwCuvvALT/9/V2c7ODkOHDsX8+fOtVtzcuXMREBCA6OhoqS0oKEj6txACS5YswZQpU9CjRw8AwMaNG+Hj44Pt27ejf//+VquFiIiIHk+l2rPj5OSElStXIiUlRTozKzU1FStXroSzc/HuG1Qc3333HZ555hm8/PLL8Pb2xtNPP421a9dKy69evYqEhAR06NBBatNoNGjWrBkOHTpU5LjZ2dlIT0+3eBAREZE8lemigvHx8YiPj0dwcDCcnZ0hhLBWXQCAK1euYNWqVQgODsauXbvwn//8B2+//TY2bNgAAEhISAAA+Pj4WKzn4+MjLStMZGQkNBqN9AgICLBq3URERFR5lCrspKSkoH379qhduzZeeOEFxMfHAwCGDh1q1dPOzWYzGjdujDlz5uDpp5/G8OHDMWzYMERFRZVp3MmTJ0On00mPuLg4K1VMRERElU2pws64ceNgb2+PGzduwMnJSWrv168fdu7cabXi/Pz8UL9+fYu2evXq4caNGwAAX19fAEBiYqJFn8TERGlZYZRKJVxdXS0eREREJE+lCjs///wz5s6di6pVq1q0BwcH4/r161YpDABatGiBS5cuWbT9/fffqF69OoC7k5V9fX2xZ88eaXl6ejqOHDmCsLAwq9VBREREj69SnY1lMBgs9ujkS01NhVKpLHNR+caNG4fmzZtjzpw56Nu3L/7880+sWbMGa9asAXD3dhRjx47F7NmzERwcjKCgIEydOhX+/v7o2bOn1eogIiKix1ep9uy0atUKGzdulJ4rFAqYzWbMmzcPbdu2tVpxzz77LL755ht8+eWXaNiwIWbNmoUlS5Zg4MCBUp9JkybhrbfewvDhw/Hss89Cr9dj586dUKlUVquDiIiIHl+l2rMzb948tG/fHseOHUNOTg4mTZqEc+fOITU1FQcPHrRqgd26dUO3bt2KXK5QKDBz5kzMnDnTqtslIiIieSjVnp2GDRvi77//RsuWLdGjRw8YDAb07t0bJ0+eRM2aNa1dIxEREVGplXjPjslkQufOnREVFYUPPvigPGoiIiIispoS79mxt7fHmTNnyqMWIiIiIqsr1WGsV199FevWrbN2LURERERWV6oJyrm5ufjss8/wyy+/oEmTJgXuh7Vo0SKrFEdERERUViUKO1euXEFgYCD++usvNG7cGMDdi/zdS6FQWK86IiIiojIqUdgJDg5GfHw89u7dC+Du7SGWLl1a4EacRERERJVFiebs3H9X8x07dsBgMFi1ICIiIiJrKtUE5Xz3hx8iIiKiyqZEYUehUBSYk8M5OkRERFSZlWjOjhACgwcPlm72mZWVhTfffLPA2VgxMTHWq5CIiIioDEoUdiIiIiyev/rqq1YthoiIiMjaShR2oqOjy6sOIiIionJRpgnKRERERJUdww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREckaww4RERHJGsMOERERyRrDDhEREcnaYxV2Pv74YygUCowdO1Zqy8rKwqhRo+Dh4QG1Wo0+ffogMTGx4ookIiKiSuWxCTtHjx7F6tWr8dRTT1m0jxs3Dt9//z22bt2K3377Dbdu3ULv3r0rqEoiIiKqbB6LsKPX6zFw4ECsXbsWbm5uUrtOp8O6deuwaNEitGvXDk2aNEF0dDT++OMPHD58uAIrJiIiosrisQg7o0aNQteuXdGhQweL9uPHj8NkMlm0161bF9WqVcOhQ4cedZlERERUCdlVdAEPs2XLFpw4cQJHjx4tsCwhIQEODg7QarUW7T4+PkhISChyzOzsbGRnZ0vP09PTrVYvERERVS6Ves9OXFwcxowZgy+++AIqlcpq40ZGRkKj0UiPgIAAq41NRERElUulDjvHjx/H7du30bhxY9jZ2cHOzg6//fYbli5dCjs7O/j4+CAnJwdpaWkW6yUmJsLX17fIcSdPngydTic94uLiyvmVEBERUUWp1Iex2rdvj7Nnz1q0DRkyBHXr1sW7776LgIAA2NvbY8+ePejTpw8A4NKlS7hx4wbCwsKKHFepVEKpVJZr7URERFQ5VOqw4+LigoYNG1q0OTs7w8PDQ2ofOnQoxo8fD3d3d7i6uuKtt95CWFgYnnvuuYoomR4hnU4Ho9FolbESExNhzMyE3mBAhlJRYLlBb4A5L88q2yIiokerUoed4li8eDFsbGzQp08fZGdnIzw8HCtXrqzosqic6XQ6fDRvMVIyrBN2jAY9ks+fxwH7OLirC+71M+XkIDklFa5Vc62yPSIienQeu7Czb98+i+cqlQorVqzAihUrKqYgqhBGoxEpGUa4N2gJtca9zONl3ElGZtJ1uPp5QKNxLrBcr0tFblIyzGZzmbf1JDGZTMi4k1ygPeNOMowGfamudu7k5ASNRmON8ojoCfHYhR2ie6k17nD18LbKWPYqFRwcnaB0Khh2crKsswfpSWLIzEb8lYvAjrWwv+9sSlNWFgxJcdi8MhVOjo4lGtfexROjJ05h4CGiYmPYIaJykW3KhUqRg+71neHtZbn3LSfTiPT4NLRs6g21c8FwWZSkNANizibDaDQy7BBRsTHsEFG50qpV8NSqLdqyHRSwy1DC110NF7VLCUfMsF5xRPREqNTX2SEiIiIqK4YdIiIikjWGHSIiIpI1hh0iIiKSNYYdIiIikjWGHSIiIpI1hh0iIiKSNYYdIiIikjWGHSIiIpI1hh0iIiKSNYYdIiIikjWGHSIiIpI1hh0iIiKSNYYdIiIikjWGHSIiIpI1u4ougKikdDodEhMTYTTokXEn2SpjZtxJRp7JZJWxiIiocmHYoceKTqfD8vmzobsdh+Tz55GZdB32KlWZx83MzET6zcvIbeZphSqJiKgyYdihx4rRaIQpIxm9GqpRz94Frn4ecHB0KvO4126lYFtcNsy5uVaokoiIKhOGHXoseWqc4K5WQqNxhtLJuczjpaYbrFAVERFVRpygTERERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLGsENERESyxrBDREREssawQ0RERLLG20UQUYXIy82FQV+y23ToDQYYMzORmJhY4u05OTlBo9GUeD0ievwx7BDRI5drykHi7UQcPHoCdvb2xV4vVZ+Nk2czELdsDZyc1SXapoeLEz6YNI6Bh+gJxLBDRI+cOS8XeUIBlVc1OKtdi71ers4AZ68UVHmmE1zcPIu9nl6XipRzB2A0Ghl2iJ5ADDtEVGEclKoS3bXeIUfAXmWAi5snXD28S7St1JIWR0SywQnKREREJGsMO0RERCRrDDtEREQkaww7REREJGsMO0RERCRrDDtEREQkaww7REREJGsMO0RERCRrDDtEREQka5U67ERGRuLZZ5+Fi4sLvL290bNnT1y6dMmiT1ZWFkaNGgUPDw+o1Wr06dOnVDcJJCIiInmq1GHnt99+w6hRo3D48GHs3r0bJpMJnTp1gsHwvzsljxs3Dt9//z22bt2K3377Dbdu3ULv3r0rsGoiIiKqTCr1vbF27txp8Xz9+vXw9vbG8ePH0bp1a+h0Oqxbtw6bN29Gu3btAADR0dGoV68eDh8+jOeee64iyiYiIqJKpFLv2bmfTqcDALi7uwMAjh8/DpPJhA4dOkh96tati2rVquHQoUNFjpOdnY309HSLBxEREcnTYxN2zGYzxo4dixYtWqBhw4YAgISEBDg4OECr1Vr09fHxQUJCQpFjRUZGQqPRSI+AgIDyLJ2IiIgq0GMTdkaNGoW//voLW7ZsKfNYkydPhk6nkx5xcXFWqJCIiIgqo0o9Zyff6NGj8cMPP2D//v2oWrWq1O7r64ucnBykpaVZ7N1JTEyEr69vkeMplUoolcryLJmIiIgqiUq9Z0cIgdGjR+Obb77Br7/+iqCgIIvlTZo0gb29Pfbs2SO1Xbp0CTdu3EBYWNijLpeIiIgqoUq9Z2fUqFHYvHkzvv32W7i4uEjzcDQaDRwdHaHRaDB06FCMHz8e7u7ucHV1xVtvvYWwsDCeiUVEREQAKnnYWbVqFQDg+eeft2iPjo7G4MGDAQCLFy+GjY0N+vTpg+zsbISHh2PlypWPuFIiIiKqrCp12BFCPLSPSqXCihUrsGLFikdQERERET1uKvWcHSIiIqKyYtghIiIiWWPYISIiIllj2CEiIiJZY9ghIiIiWWPYISIiIllj2CEiIiJZY9ghIiIiWWPYISIiIllj2CEiIiJZY9ghIiIiWWPYISIiIllj2CEiIiJZY9ghIiIiWWPYISIiIllj2CEiIiJZY9ghIiIiWbOr6AKIiErCZDIh405yidbJuJMMo0GPxMTEQpc7OTlBo9FYozwiqoQYdojosWHIzEb8lYvAjrWwV6mKvZ4pKwuGpDhsXpkKJ0fHAsvtXTwxeuIUBh4imWLYoXKl0+lgNBqtNl5iYiJyTCarjUePl2xTLlSKHHSv7wxvL/dir5eTaUR6fBpaNvWG2tnZYllSmgExZ5NhNBoZdohkimGHyo1Op8Py+bNhyijZIYcHyTAYceXv88h+urnVxqTHj1atgqdWXez+2Q4K2GUo4euuhovapZAeGdYrjogqHYYdKjdGoxGmjGT0DnGBl9b54SsUw/lrt7HsXDbyTHlWGY+IiOSPYYfKnZfWGX4erlYZK/GO3irjEBHRk4OnnhMREZGscc8OWc39k5ETExNhzMyE3mBAhlJhlW0YjEbk5uXCmGmEOY+HsoiI6OEYdsgqdDodPpq3GCkZ/ws7RoMeyefP44B9HNzVSqts50piBlLu6HDqr/NQ5OXAtWquVcYlIiL5YtghqzAajUjJMMK9QUuoNXdPCc64k4zMpOtw9fOARmOdCcouSIK9KgEOWh9kJl2H2Wy2yrhERCRfDDtkVWqNO1w9vKXn9ioVHBydoHSyTtixV+mhsLWFvb0DMq0yIhERyR0nKBMREZGsMewQERGRrDHsEBERkawx7BAREZGsMewQERGRrDHsEBERkawx7BAREZGsMewQERGRrPGigkT0xMvKzkFiYqLVx3VycoJGo7H6uERUMgw7RPREyMvNhUFvKNAen5SKEyeOw7hwOlSOjlbbnp2dHZzd/TB64hQGHqIKxrBDRLKXa8pB4u1EHDx6Anb29hbLriRmIONOEqoLR7jbO1ltmzkmgeupd+8bx7BDVLEYdohI9sx5ucgTCqi8qsFZ7WqxLP/msv5BteDn7WmV7eVkZ+Lm1VjkmnKtMh4RlQ3DDhE9MRyUqgI3pc2/uayD0tFqN6wlosqFZ2MRERGRrDHsEBERkazxMBZJdDodjEZjqdZNTEyE0aBHxp1kqS3jTjLyTCZrlUdERFQqsgk7K1aswPz585GQkIDQ0FAsW7YMTZs2reiyKlRJwkt6ejrWRy1BnuFOqbaVlZmJ23//g8yk67BXqQAAmZmZSL95GbnNrDPpk4iIqDRkEXa++uorjB8/HlFRUWjWrBmWLFmC8PBwXLp0Cd7e3hVdXoXQ6XT4aN5ipGQUL+wYDXoknz+IbnVVcHWyf/gK98mFCVUCchAUooWj2gUAcO1WCrbFZcOcyzNS6MmUnWP9ixXyQoVEJSeLsLNo0SIMGzYMQ4YMAQBERUXhxx9/xGeffYb33nuvgquzvuLssUlMTERcwm1o6zSFs4v2oWPa61KhT/gHVWv6wkNT8jNS9LpUZBvT4eaiglqjBgCkphe8gBvRk0JvzMbZM2fwmZUvVmjr7IbBb46Fq6vlKfQMQURFe+zDTk5ODo4fP47JkydLbTY2NujQoQMOHTpUgZXdVZZ5MIVJT0/H55+ugG1OxgP7GTMzkXz+vMVhpQfJzMxEevx12NhVLdXptzlZ1nuNRI+7XFMO4hOTYNSlo7q4YbWLFaYbTdh+wIALCUY43vd76qZ2xJiRwwuEoOIwmUywty/5Ht2HYQCjyuKxDzvJycnIy8uDj4+PRbuPjw8uXrxY6DrZ2dnIzs6Wnut0OgB3g4Q16XQ6rFk6H7kZKVYbM8OYiWv/XMSQTk/BS1v0l5oxywRthh0ctHmwd3j4YaSElCz8dlmPa3Hx0OtLHlyMeh0S03Pg8O9tOKbe3aPz7+00mHJMuBGfBGNWTonHLEz+mDeT7sB43/asMW5RtRb2+so65oMUtb3yeE8B4FZSGvLMAjeTUgE7pVXGfFCtpXk/Hzbmgzxoe+Xxnhr1OqQYTMizsUOeoxZ5TiUPIIUxZBuQqouH4uYlGB3+9znlmUy4ZUjF4oQLUJZwL1J2Tg5iL19DnVpBVg88tk5aDBw6Ei4uLmUeSwgBhUJhhaq4vYrYnrOzs1V+Du6X/3dbCPHgjuIxd/PmTQFA/PHHHxbtEydOFE2bNi10nenTpwsAfPDBBx988MGHDB5xcXEPzAqP/Z4dT09P2NraFpgEmJiYCF9f30LXmTx5MsaPHy89N5vNSE1NhYeHxyNNuhUpPT0dAQEBiIuLK9Vubyo//GwqL342lRc/m8qrPD8bIQQyMjLg7+//wH6PfdhxcHBAkyZNsGfPHvTs2RPA3fCyZ88ejB49utB1lEollErLXfVarbacK62cXF1d+cVQSfGzqbz42VRe/Gwqr/L6bIozL+yxDzsAMH78eEREROCZZ55B06ZNsWTJEhgMBunsLCIiInpyySLs9OvXD0lJSZg2bRoSEhLQqFEj7Ny5s8CkZSIiInryyCLsAMDo0aOLPGxFBSmVSkyfPr3A4TyqePxsKi9+NpUXP5vKqzJ8NgohHna+FhEREdHji3c9JyIiIllj2CEiIiJZY9ghIiIiWWPYISIiIllj2JGB/fv3o3v37vD394dCocD27dsf2D8+Ph6vvPIKateuDRsbG4wdO7ZAn+effx4KhaLAo2vXruXzImSqPD4bAFiyZAnq1KkDR0dHBAQEYNy4ccjKyrL+C5Cx8vhsTCYTZs6ciZo1a0KlUiE0NBQ7d+4snxcgYyX9bGJiYtCxY0d4eXnB1dUVYWFh2LVrV4F+K1asQGBgIFQqFZo1a4Y///yznF6BfJXHZ1PSMUuDYUcGDAYDQkNDsWLFimL1z87OhpeXF6ZMmYLQ0NBC+8TExCA+Pl56/PXXX7C1tcXLL79szdJlrzw+m82bN+O9997D9OnTceHCBaxbtw5fffUV3n//fWuWLnvl8dlMmTIFq1evxrJly3D+/Hm8+eab6NWrF06ePGnN0mWvpJ/N/v370bFjR/z00084fvw42rZti+7du1u871999RXGjx+P6dOn48SJEwgNDUV4eDhu375dXi9DlsrjsynpmKVindtxUmUBQHzzzTfF7t+mTRsxZsyYh/ZbvHixcHFxEXq9vvTFPeGs9dmMGjVKtGvXzqJt/PjxokWLFmWs8Mllrc/Gz89PLF++3KKtd+/eYuDAgWWs8MlV0s8mX/369cWMGTOk502bNhWjRo2Snufl5Ql/f38RGRlpjTKfSNb6bKwx5sNwzw4Vy7p169C/f384OztXdClPvObNm+P48ePSLvgrV67gp59+wgsvvFDBlVF2djZUKpVFm6OjIw4cOFBBFT2ZzGYzMjIy4O7uDgDIycnB8ePH0aFDB6mPjY0NOnTogEOHDlVUmU+k+z+bR0U2V1Cm8vPnn3/ir7/+wrp16yq6FALwyiuvIDk5GS1btoQQArm5uXjzzTd5GKsSCA8Px6JFi9C6dWvUrFkTe/bsQUxMDPLy8iq6tCfKggULoNfr0bdvXwBAcnIy8vLyCtxCyMfHBxcvXqyIEp9Y9382jwr37NBDrVu3DiEhIWjatGlFl0IA9u3bhzlz5mDlypU4ceIEYmJi8OOPP2LWrFkVXdoT75NPPkFwcDDq1q0LBwcHjB49GkOGDIGNDb9qH5XNmzdjxowZ+Prrr+Ht7V3R5dA9KvKz4W8gPZDBYMCWLVswdOjQii6F/t/UqVPx2muv4Y033kBISAh69eqFOXPmIDIyEmazuaLLe6J5eXlh+/btMBgMuH79Oi5evAi1Wo0aNWpUdGlPhC1btuCNN97A119/bXHIytPTE7a2tkhMTLTon5iYCF9f30dd5hOpqM/mUWHYoQfaunUrsrOz8eqrr1Z0KfT/jEZjgT0Ftra2AADBW91VCiqVClWqVEFubi7++9//okePHhVdkux9+eWXGDJkCL788ssCl8hwcHBAkyZNsGfPHqnNbDZjz549CAsLe9SlPnEe9Nk8KpyzIwN6vR7//POP9Pzq1as4deoU3N3dUa1aNUyePBk3b97Exo0bpT6nTp2S1k1KSsKpU6fg4OCA+vXrW4y9bt069OzZEx4eHo/ktchNeXw23bt3x6JFi/D000+jWbNm+OeffzB16lR0795dCj30cOXx2Rw5cgQ3b95Eo0aNcPPmTXz44Ycwm82YNGnSI31tj7uSfjabN29GREQEPvnkEzRr1gwJCQkA7k4O12g0AIDx48cjIiICzzzzDJo2bYolS5bAYDBgyJAhj/4FPsbK47N52JhWYfXzu+iR27t3rwBQ4BERESGEECIiIkK0adPGYp3C+levXt2iz8WLFwUA8fPPPz+aFyJD5fHZmEwm8eGHH4qaNWsKlUolAgICxMiRI8WdO3ce2euSg/L4bPbt2yfq1asnlEql8PDwEK+99pq4efPmo3tRMlHSz6ZNmzYP7J9v2bJlolq1asLBwUE0bdpUHD58+NG9KJkoj8/mYWNag0II7vcmIiIi+eKcHSIiIpI1hh0iIiKSNYYdIiIikjWGHSIiIpI1hh0iIiKSNYYdIiIikjWGHSIiIpI1hh0iqnAKhQLbt2+v6DKISKYYdoio3CUkJOCtt95CjRo1oFQqERAQgO7du1vcq4iIqLzw3lhEVK6uXbuGFi1aQKvVYv78+QgJCYHJZMKuXbswatQoXLx4saJLJCKZ454dIipXI0eOhEKhwJ9//ok+ffqgdu3aaNCgAcaPH4/Dhw8Xus7Zs2fRrl07ODo6wsPDA8OHD4der5eW79u3D02bNoWzszO0Wi1atGiB69evS8u//fZbNG7cGCqVCjVq1MCMGTOQm5srLVcoFPj000/Rq1cvODk5ITg4GN99951FDX/99Re6dOkCtVoNHx8fvPbaa0hOTpaWb9u2DSEhIVKNHTp0gMFgKFZ9RPRoMewQUblJTU3Fzp07MWrUKDg7OxdYrtVqC7QZDAaEh4fDzc0NR48exdatW/HLL79g9OjRAIDc3Fz07NkTbdq0wZkzZ3Do0CEMHz4cCoUCAPD7779j0KBBGDNmDM6fP4/Vq1dj/fr1+Oijjyy2M2PGDPTt2xdnzpzBCy+8gIEDByI1NRUAkJaWhnbt2uHpp5/GsWPHsHPnTiQmJqJv374AgPj4eAwYMACvv/46Lly4gH379qF3794QQjy0PiKqAFa7pSgR0X2OHDkiAIiYmJgH9gMgvvnmGyGEEGvWrBFubm5Cr9dLy3/88UdhY2MjEhISREpKigAg9u3bV+hY7du3F3PmzLFo27Rpk/Dz87PY3pQpU6Tner1eABA7duwQQggxa9Ys0alTJ4sx4uLiBABx6dIlcfz4cQFAXLt2rcD2H1YfET163LNDROVGCFHidS5cuIDQ0FCLPUEtWrSA2WzGpUuX4O7ujsGDByM8PBzdu3fHJ598gvj4eKnv6dOnMXPmTKjVaukxbNgwxMfHw2g0Sv2eeuop6d/Ozs5wdXXF7du3pTH27t1rMUbdunUBAJcvX0ZoaCjat2+PkJAQvPzyy1i7di3u3LkDAA+tj4gePYYdIio3wcHBUCgUVp+EHB0djUOHDqF58+b46quvULt2bWn+j16vx4wZM3Dq1CnpcfbsWcTGxkKlUklj2NvbW4ypUChgNpulMbp3724xxqlTpxAbG4vWrVvD1tYWu3fvxo4dO1C/fn0sW7YMderUwdWrVx9aHxE9egw7RFRu3N3dER4ejhUrVkiTd++VlpZWoK1evXo4ffq0Rf+DBw/CxsYGderUkdqefvppTJ48GX/88QcaNmyIzZs3AwAaN26MS5cuoVatWgUeNjbF+8pr3Lgxzp07h8DAwAJj5O9xUigUaNGiBWbMmIGTJ0/CwcEB33zzzUPrI6JHj2GHiMrVihUrkJeXh6ZNm+K///0vYmNjceHCBSxduhRhYWEF+g8cOBAqlQoRERH466+/sHfvXrz11lt47bXX4OPjg6tXr2Ly5Mk4dOgQrl+/jp9//hmxsbGoV68eAGDatGnYuHEjZsyYgXPnzuHChQvYsmULpkyZUuyaR40ahdTUVAwYMABHjx7F5cuXsWvXLgwZMgR5eXk4cuQI5syZg2PHjuHGjRuIiYlBUlIS6tWr99D6iKgCVPSkISKSv1u3bolRo0aJ6tWrCwcHB1GlShXx4osvir179wohLCcoCyHEmTNnRNu2bYVKpRLu7u5i2LBhIiMjQwghREJCgujZs6fw8/MTDg4Oonr16mLatGkiLy9PWn/nzp2iefPmwtHRUbi6uoqmTZuKNWvWSMvv354QQmg0GhEdHS09//vvv0WvXr2EVqsVjo6Oom7dumLs2LHCbDaL8+fPi/DwcOHl5SWUSqWoXbu2WLZsWbHrI6JHSyFEKWYQEhERET0meBiLiIiIZI1hh4iIiGSNYYeIiIhkjWGHiIiIZI1hh4iIiGSNYYeIiIhkjWGHiIiIZI1hh4iIiGSNYYeIiIhkjWGHiIiIZI1hh4iIiGSNYYeIiIhk7f8AJ4UW2ljM2ZEAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1.18540708, 1.18084023, 1.18320454, ..., 1.17846852, 1.18315814,\n",
              "       1.18232426])"
            ]
          },
          "metadata": {},
          "execution_count": 130
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Note:\n",
        "I find it unlikely that such a simple model as this Bradley-Terry model can seperate these two gaussians. There is no appreciable difference in distance between queries which require the same model and queries which require a different model.\n",
        "\n",
        "I do think a model could be trained to embed this information, something like a CLIP style model perhaps, but using an off the shelf general encoder for this task seems dubious.\n",
        "\n",
        "So, I'm going to move onto the other method."
      ],
      "metadata": {
        "id": "or2FF5zRtJ6C"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "XlFWyjeQrMbY"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}